Insight Global is seeking an experienced Senior Data Engineer to join our client's data modernization project. This role is pivotal in transitioning data from on-premises infrastructure to a cloud-based environment on Azure. As a Senior Data Engineer, you will leverage your expertise in SQL, PySpark, and Python to facilitate data transformation and migration tasks. You will also play a key role in managing and optimizing data workflows using Databricks and Snowflake, ensuring data security and compliance throughout the process.
Key Responsibilities
- Data Migration & Transformation. Utilize SQL, PySpark, and Python to support data transformation and migration from on-premises to Azure cloud infrastructure.
- Workflow Management. Manage and optimize data workflows using Azure Databricks and Snowflake.
- Batch Process Support. Provide support for batch processes, ensuring timely and efficient data handling.
- Data Security & Compliance. Ensure data security and compliance throughout the migration process, adhering to best practices and regulatory requirements.
- Collaboration. Work closely with cross-functional teams to align data engineering efforts with project goals and client needs.
Required Skills and Experience
- Azure Databricks & Data Factory. Proven experience with Azure Databricks and Data Factory for managing data pipelines and workflows.
- SQL. Strong proficiency in SQL for data querying and manipulation.
- PySpark or Spark. Expertise in PySpark or Spark for large-scale data processing.
- Snowflake. Experience with Snowflake for cloud data warehousing and analytics.
- PowerBI (Optional). Knowledge of PowerBI is a plus, though not required.
- Experience. Minimum of 5 years in a data engineering role with a solid understanding of data engineering principles.
- Batch Processes. Experience in supporting and managing batch processes.
Ideal Candidate
The ideal candidate will have a robust background in data engineering with a focus on cloud infrastructure and batch process support. Strong problem-solving skills, attention to detail, and the ability to work effectively in a remote team environment are essential.
Why Join Us?
- Remote Flexibility. Work from anywhere in India while collaborating with a dynamic team.
- Impactful Projects. Contribute to a major data modernization project that shapes the future of cloud data infrastructure.
- Professional Growth. Enhance your skills and career prospects with hands-on experience in cutting-edge technologies.
Apply today to be a part of this exciting data transformation journey!