We are seeking an experienced and visionary Senior Data Engineer to lead our dynamic team of data professionals. In this pivotal role, you will be responsible for driving the strategy, roadmap, and execution of key data engineering initiatives. You will collaborate with stakeholders across the organization to understand their data needs, create robust data models, and develop scalable data solutions. Your expertise will ensure the successful delivery of high-quality data pipelines and data management systems.
Key Responsibilities
- Leadership and Mentoring. Lead and mentor a team of data engineers. Coordinate with other teams to manage and prioritize projects, driving strategy, roadmap, and execution of data engineering initiatives.
- Requirement Gathering and Analysis. Understand and translate business needs into comprehensive data models. Work with business teams to implement data strategies, build data flows, and develop conceptual data models.
- Data Pipeline Design and Development. Create robust and scalable data pipelines and data products. Develop and maintain data lakes, integrating data from primary and secondary sources to enhance evaluation processes.
- Data Integration and Management. Develop and maintain data lakes, build flexible and scalable scripts for data processing, and ensure adherence to defined business rules and quality thresholds.
- Testing and Deployment. Define and manage data load procedures, ensuring data quality and adherence to business rules. Collaborate with development teams to implement data strategies and develop physical data models.
- Continuous Integration and Deployment (CI/CD). Implement and maintain CI/CD pipelines for data solutions, ensuring rapid, reliable, and streamlined updates to the data environment.
- Stakeholder Collaboration. Work closely with stakeholders to understand their data needs and deliver tailored solutions. Communicate data warehouse plans effectively to internal clients.
Qualifications
Technical Skills
- Extensive experience with big data tools such as Hadoop, Hive, and Spark.
- Proficiency in Scala, Python, SQL, and PySpark.
- Experience with Unix/Linux systems and scripting in Bash.
- Familiarity with data pipeline and workflow management tools like Airflow.
- Experience with relational SQL and NoSQL databases, including Postgres and Cassandra.
- Proficiency in cloud services such as AWS or Azure.
- Experience with stream-processing systems like Kafka and Spark-Streaming.
- Strong analytical skills related to structured and unstructured datasets.
- Experience in managing and communicating data warehouse plans.
Additional Skills
- Strong problem-solving abilities and excellent communication skills.
- Detail-oriented with exceptional organizational skills.
- Proven leadership experience in managing data engineering teams.
Experience and Education
- 8+ years of relevant work experience with a bachelor’s degree or 5+ years with an advanced degree (Master’s, MBA, JD, MD) or 3+ years with a PhD.
- Exposure to the Financial Services/Payments industry is a plus.
Leadership Competencies
- Exhibits intellectual curiosity and a commitment to continuous learning.
- Demonstrates integrity, maturity, and a constructive approach to business challenges.
- Acts as a role model for the organization, embodying core values.
- Respects individuals at all levels and strives for excellence.
- Uses sound insights and judgment to make informed decisions aligned with business strategy.
- Possesses strong leadership skills, including the ability to allocate resources and influence senior management.