We are seeking a highly skilled Data Engineer with over 5 years of hands-on experience in Big Data technologies to join our team. The ideal candidate will have a strong background in Python and PySpark, with proven expertise in optimizing Spark jobs for processing large volumes of data. You will play a critical role in managing and transforming data using various AWS services and tools.
Key Responsibilities
- Big Data Processing. Design, implement, and optimize data pipelines using PySpark to handle and process large-scale data efficiently.
- AWS Services Integration. Utilize Amazon EMR, Amazon Athena, AWS Glue, Amazon Lambda, Amazon EC2, Amazon S3, and SNS for data storage, processing, and integration tasks.
- Optimization. Analyze and optimize Spark jobs for performance and scalability, ensuring efficient data processing.
- Version Control. Maintain code integrity and version control using GIT, collaborating with team members to manage and track changes.
- Collaboration. Work closely with data scientists, analysts, and other stakeholders to understand data requirements and provide solutions that meet business needs.
Required Skills and Qualifications
- Experience. Minimum of 5 years of relevant experience in Data Engineering, with hands-on expertise in Big Data technologies.
- Python and PySpark. Proficiency in Python and PySpark for data processing and analytics.
- AWS Expertise. In-depth experience with Amazon EMR, Amazon Athena, AWS Glue, Amazon Lambda, Amazon EC2, Amazon S3, and SNS.
- Optimization Skills. Proven ability to optimize Spark jobs for handling large volumes of data.
- Version Control. Hands-on experience with GIT for version control and collaboration.
- Problem-Solving. Strong analytical and problem-solving skills with a focus on data integrity and performance.
Desired Attributes
- Team Player. Excellent communication skills and ability to work collaboratively in a team environment.
- Adaptability. Ability to adapt to changing requirements and technologies in a fast-paced environment.
- Detail-Oriented. Attention to detail and commitment to delivering high-quality results.
Why Join Us?
- Innovative Projects. Work on cutting-edge projects that leverage the latest Big Data technologies and AWS services.
- Growth Opportunities. Access to professional development opportunities and career growth within a dynamic team.
- Collaborative Culture. Join a team of talented professionals who value collaboration, innovation, and excellence.
Application Process
If you meet the above qualifications and are passionate about data engineering, we encourage you to apply. Please submit your resume and cover letter outlining your relevant experience and skills.