We are seeking a highly skilled and experienced Azure Data Engineer to join our dynamic team. This individual will be responsible for designing, developing, and maintaining robust ETL/ELT pipelines using Azure Data Factory, working with large-scale data processing systems, and collaborating with business stakeholders to deliver efficient and scalable data solutions. The ideal candidate will have at least 6 years of hands-on experience in building data solutions in the Azure cloud environment, with expertise in data ingestion, transformation, and loading from a variety of sources.
Key Responsibilities
- Design and Develop ETL/ELT Pipelines. You will play a pivotal role in developing end-to-end data workflows utilizing Azure Data Factory. This includes ingesting data from diverse sources, transforming it into meaningful insights, and loading it into target systems. Your ability to write optimized and efficient SQL queries will be crucial in manipulating and aggregating large datasets from relational databases like Azure SQL and Synapse Analytics.
- Data Storage Solutions. You will be responsible for implementing data storage solutions within Azure Data Lake, ensuring that data is efficiently structured, partitioned, and access-controlled. This involves making sure the data is accessible to those who need it while maintaining the necessary security and performance requirements.
- Data Processing with PySpark and Python. The role requires a deep understanding of distributed data processing systems. You will leverage PySpark and Python to develop complex data transformations that ensure high performance and scalability, enabling large-scale data operations on cloud platforms. Your coding expertise will be essential in optimizing and streamlining data workflows.
- Ensuring Data Quality and Consistency. We are committed to providing clean, accurate, and reliable data. You will build robust error handling, logging mechanisms, and data validation checks within the ETL pipelines to ensure that our data is of the highest quality, meeting the needs of downstream applications and analytics.
- Collaboration with Data Architects and Business Stakeholders. As an integral member of the team, you will work closely with data architects and business stakeholders to understand data requirements and business needs. This will allow you to build data models and design optimal data solutions that support business objectives and provide actionable insights.
- Pipeline Monitoring and Optimization. In addition to development, you will monitor and troubleshoot data pipelines, ensuring they operate smoothly and efficiently. This includes performance tuning, identifying bottlenecks, and optimizing pipeline performance to enhance data availability and cost-effectiveness. You will also be responsible for maintaining high levels of uptime for critical data services.
- Implementing Data Security and Governance Practices. You will be responsible for ensuring that data storage, processing, and access in the cloud adhere to best practices in data security, governance, and compliance. This includes ensuring that sensitive data is encrypted, access controls are enforced, and the data complies with industry regulations.
- DevOps and CI/CD Integration. Working as part of a modern engineering team, you will actively participate in DevOps practices, helping automate and streamline the deployment process through CI/CD pipelines using Azure DevOps or similar tools. This will help ensure the continuous integration and deployment of data solutions, reducing deployment time and minimizing errors.
Required Skills and Experience
- At least 6 years of hands-on experience as a Data Engineer with expertise in Azure Data Factory for building ETL/ELT pipelines.
- Strong proficiency in PySpark and Python, with experience in handling distributed data processing in cloud environments.
- In-depth experience with SQL and writing optimized queries for large datasets in Azure SQL, Synapse Analytics, or other relational systems.
- Solid understanding of Azure Data Lake and experience in managing data storage solutions, including partitioning, access controls, and structuring large datasets for efficient querying.
- Experience with building robust data quality frameworks, error handling, logging, and data validation within ETL processes.
- Strong collaboration skills, with the ability to engage with data architects and business stakeholders to understand data requirements and design optimal solutions.
- Experience with monitoring and troubleshooting data pipelines for performance issues, and a proactive approach to identifying and resolving data pipeline inefficiencies.
- Solid understanding of data security and governance best practices, ensuring compliance with industry standards and organizational policies.
- Familiarity with DevOps practices, particularly around deploying data solutions using CI/CD pipelines and Azure DevOps or similar tools.
Preferred Qualifications
- Experience working with big data technologies and frameworks such as Hadoop, Spark, and Databricks.
- Knowledge of Azure Synapse Analytics, Azure Databricks, or similar advanced analytics tools within the Azure ecosystem.
- Familiarity with additional programming languages, such as Scala or Java, is a plus.
- Strong problem-solving skills and an ability to work independently in a fast-paced, collaborative environment.
- A degree in Computer Science, Engineering, Data Science, or a related field.
Why Join Us?
Joining our team means being part of an innovative, forward-thinking company that values creativity, collaboration, and continuous learning. You will have the opportunity to work on cutting-edge data technologies in a fast-growing industry while making a significant impact on data-driven decision-making across the organization.
If you're passionate about data engineering, cloud technologies, and optimizing data pipelines for high-performance systems, we would love to hear from you. Apply now and take the next step in your career with us!