We are seeking an experienced Senior Data Engineer with a robust background in Data Warehouse and Hadoop platforms. This role requires 10+ years of overall experience in data engineering, with a deep understanding of Big Data environments, particularly in leveraging Python, PySpark, and Hive. The ideal candidate will possess advanced SQL skills and a strong foundation in Data Warehouse concepts. Experience with migration from legacy data platforms to Hadoop and familiarity with AWS or other cloud platforms are highly desirable.
Key Responsibilities
- Data Engineering and Development. Design, build, and optimize scalable and efficient data pipelines on Hadoop and Data Warehouse platforms using Python/PySpark, ensuring data is accessible and secure.
- Complex SQL Development. Write and optimize complex SQL queries to extract, transform, and load (ETL) data from various sources, supporting analytics, reporting, and data-driven insights.
- Data Warehousing Concepts. Apply in-depth knowledge of Data Warehouse principles, including data modeling, normalization, and denormalization, to design robust and reliable data solutions.
- Data Migration. Lead and execute data migration initiatives from legacy Data Warehouse platforms to Hadoop, ensuring data integrity, efficiency, and minimal disruption.
- Cloud Platform Exposure. Utilize AWS or other cloud platforms to enhance data storage, processing, and deployment capabilities, supporting the organization’s cloud strategy.
Required Skills
- Python/PySpark. Proficiency in Python or Spark for data manipulation and transformation in a Big Data environment.
- Hive. Strong experience in Hive for data query and analysis.
- SQL Proficiency. Ability to write and optimize complex SQL queries, with experience handling large datasets.
- Data Warehouse Concepts. Solid understanding of Data Warehouse architecture, ETL processes, and data modeling.
- Migration Experience. Exposure to data migration from legacy systems to Hadoop platforms is a significant plus.
- Cloud Platform Knowledge. Familiarity with AWS or other cloud environments is highly advantageous.
Qualifications
- Bachelor’s or Master’s degree in Computer Science, Information Technology, or a related field.
- 10+ years of experience in Data Engineering, with a focus on Data Warehouse and Hadoop environments.
- Proven track record of delivering data solutions in a complex, high-volume environment.
Ideal Candidate Profile
- A data enthusiast who can handle complex data transformations, migrations, and optimizations.
- Strong analytical skills with a focus on delivering actionable insights.
- Familiarity with cloud platforms like AWS is a plus, helping us future-proof our data solutions.
Why Join Us?
This is an opportunity to work with a forward-thinking team, using cutting-edge data technologies. Be part of a dynamic environment where your expertise in Python, PySpark, SQL, and Data Warehousing will make a tangible impact.