We are seeking an experienced Data Engineer to design, develop, and deliver ADF pipelines across various integration points for our Accounting & Reporting Stream. The ideal candidate will have extensive experience in Azure Databricks and related technologies, with a strong background in data engineering and architecture.
Role and Responsibilities
- Pipeline Development. Design and implement scalable data pipelines using PySpark and ETL workflows with Azure Databricks and Azure Data Factory.
- Data Modeling & Architecture. Build and optimize data models and structures that support analytics and meet business requirements.
- Performance Optimization. Monitor, tune, and troubleshoot pipeline performance to ensure efficiency and reliability.
- Collaboration. Partner with business analysts and stakeholders to understand data needs and deliver actionable insights.
- Data Governance. Implement data governance practices to ensure data quality, security, and compliance with regulations.
- Documentation. Develop and maintain comprehensive documentation for data pipelines and architecture.
- Testing. Conduct testing and automation to ensure data integrity and performance.
- Cross-Functional Collaboration. Work closely with cross-functional teams to understand data requirements and provide technical guidance.
Qualifications
- Experience Level. Minimum of 4 years of extensive experience in Azure Databricks, Unity Catalog, Azure Data Factory, and Azure DevOps. Experience in the insurance domain is desirable.
- Education Level. Bachelor’s or Master’s degree in Computer Science, Information Technology, Data Science, or a related field.
- Certifications. Azure Data Engineer Associate or similar certifications, with experience in additional Azure services such as Azure Synapse Analytics and Azure SQL Database.
Mandatory Skills
- Technical Expertise. Strong background in data engineering, with experience architecting complex data systems in large-scale environments. Proficiency in SQL, Azure Databricks, Blob Storage, Azure Data Factory, and programming languages such as Python or Scala. Familiarity with Logic App and Key Vault is a plus.
- Analytical Skills. Strong analytical skills and the ability to solve complex problems effectively.
- Communication Skills. Excellent communication abilities, with a knack for explaining complex technical concepts to non-technical stakeholders.
Personal Attributes
- Leadership capabilities, effective communication skills, and a proactive problem-solving mindset are essential for success in this role.