We are seeking an experienced Senior Data Engineer to join our dynamic team. This role requires a deep understanding of data architecture and hands-on expertise with Azure Cloud and PySpark. The ideal candidate will be skilled in architecting and implementing data solutions, with a proven track record in ETL processes and data governance.
Experience. 11+ years
Key Responsibilities
- Data Architecture. Design and implement robust data architectures using Azure Data Factory (ADF) and Azure Data Lake Storage (ADLS).
- ETL Development. Develop and maintain ETL pipelines, ensuring data accuracy and efficiency.
- Azure Cloud. Leverage Azure services such as ADLS, Delta Lake, Delta Live Tables, Delta Sharing, and Unity Catalog for comprehensive data solutions.
- Data Cataloguing & Governance. Apply best practices in data cataloguing and governance, with a minimum of 6 months hands-on experience.
- Distributed Data Processing. Utilize Spark and Databricks with Python for distributed data processing tasks.
- Programming. Write efficient Python and PySpark code for data manipulation and processing.
- SQL Scripting. Perform CRUD operations on time series databases and optimize SQL queries.
- Testing. Conduct unit testing and system integration testing using Python frameworks.
- CI/CD Pipelines. Design and deploy CI/CD pipelines using Jenkins, integrating with DevOps practices.
- Agile Practices. Collaborate within high-performance agile teams, using Scrum, JIRA, JFrog, and Confluence.
Technical Skills (Must-Have)
- Distributed Data Processing. Hands-on experience with Spark and Databricks.
- Programming Languages. Proficiency in Python and PySpark.
- Azure Services. Expertise in ADLS, Delta Lake, Delta Live Tables, Delta Sharing, and Unity Catalog.
- SQL. Experience with SQL scripting and CRUD operations on time series databases.
- Testing. Applied experience in unit testing and system integration testing using Python.
- DevOps. Experience in designing and deploying CI/CD pipelines using Jenkins.
- Certifications. AZURE Data Engineering (DP203) or Databricks certification.
- Agile Tools. Familiarity with Scrum, JIRA, JFrog, and Confluence.
Preferred Qualifications
- Domain Experience. Experience in the Medical/Healthcare domain is a plus.
- Communication Skills. Excellent communication skills are essential for collaborating with cross-functional teams and stakeholders.
Why Join Us?
- Innovative Environment. Work in a cutting-edge environment where you can drive significant impact.
- Career Growth. Opportunities for professional development and career advancement.
- Dynamic Team. Collaborate with a high-performance team committed to excellence.
If you are a seasoned Data Engineer with a passion for data architecture and cloud technologies, we would love to hear from you. Apply now to join our team and be part of exciting data-driven projects!