We are seeking an experienced Senior Big Data Engineer to join our team. The ideal candidate will possess over 6 years of hands-on experience in Big Data technologies and a strong background in Java/Python. As part of our data engineering team, you will work on cutting-edge solutions to handle and process large-scale data, providing valuable insights for our business and clients. This role offers an excellent opportunity to work with modern technologies like Hadoop, Spark, Hive, and more.
Key Responsibilities
- Design, develop, and implement Big Data solutions for large-scale data processing.
- Utilize Apache Hadoop, Spark, Hive, Impala, and Yarn to process and analyze data.
- Work on data querying and analysis, using tools like Hue and Talend for efficient data processing and transformation.
- Collaborate with cross-functional teams to understand data requirements and create efficient data pipelines.
- Perform data extraction, transformation, and loading (ETL) to support data warehouse needs.
- Optimize and manage large datasets in SQL databases and perform complex queries.
- Write and maintain UNIX shell scripts for data processing automation and integration tasks.
- Lead Big Data projects, ensuring high-quality deliverables and meeting project deadlines.
Mandatory Skills
- Big Data Expertise (6+ years). Proven experience in handling large data sets and managing data pipelines.
- Java/Python (J2EE). Strong programming skills with either Java or Python, used to build scalable data solutions.
- Apache Hadoop. Extensive experience working with Hadoop frameworks and ecosystem.
- Apache Spark. Strong experience in distributed data processing using Spark.
- Apache Hive. Expertise in managing and querying data in Hive.
- Impala & YARN. Knowledge of Impala for fast, distributed SQL query execution and YARN for resource management.
- SQL Databases. Proficiency in relational database systems and writing advanced SQL queries.
- UNIX Shell Scripting. Expertise in writing shell scripts for data automation and scheduling tasks.
- Talend & Hue. Experience using Talend for data integration and Hue for interactive SQL querying and data analysis.
Preferred Qualifications
- Familiarity with Big Data reporting and visualization tools.
- Experience in query optimization and performance tuning.
- Ability to work independently and as part of a team in a fast-paced environment.
- Strong problem-solving skills and ability to debug complex issues in data pipelines.
- Excellent communication skills, both written and verbal.
What We Offer
- Opportunity to work on exciting and innovative Big Data projects.
- Competitive salary and benefits package.
- A dynamic and collaborative work environment.
- Continuous learning and development opportunities.
Join us to leverage your Big Data expertise and contribute to shaping data-driven solutions!