Insight Global is seeking a highly skilled Data Engineer to join the product development team of a consulting firm based in Chicago, Illinois. This role offers a unique opportunity to work on a healthcare application, with a focus on managing and optimizing data lakes and architecture within an AWS environment. The ideal candidate will possess 7-10+ years of hands-on experience in data engineering, with a deep understanding of AWS data lake architecture, data migration, and ETL processes. If you're passionate about creating scalable and innovative data solutions, this position is perfect for you.
Key Responsibilities
- Design and Build AWS Data Lake Architecture. Take the lead in designing and delivering innovative solutions for AWS data lake architecture. You'll be responsible for crafting scalable and efficient data architectures that cater to complex healthcare data needs.
- AWS Data Migration. Set up, configure, and troubleshoot AWS data migration services. You will ensure seamless migration of large datasets while maintaining data integrity and security throughout the process.
- Glue and DBT Configuration. Work extensively with AWS Glue or DBT for ETL (Extract, Transform, Load) operations. You will be responsible for cataloging data, setting up, configuring, and troubleshooting Glue or DBT jobs to ensure high-performance data transformation processes.
- ETL Operations. Conduct comprehensive ETL work to manage large datasets. You’ll design and implement efficient ETL pipelines to integrate and transform data from various sources into the data lake for further analysis.
- Hudi Database Management. Manage tables and data within a Hudi database, ensuring optimal performance, data consistency, and storage efficiency. You will be responsible for setting up and maintaining data tables, managing partitioning schemes, and troubleshooting any issues that arise.
- Event Bridge Scheduler and SQS Development. Develop and manage solutions using AWS SQS (Simple Queue Service) to ensure smooth data flow. Additionally, you’ll work with Event Bridge Scheduler to automate scheduling and event-driven architectures, helping to maintain seamless data operations.
- RedShift Optimization. Create and manage tables, views, and materialized views in AWS RedShift, optimizing their performance for analytical queries. You’ll work on writing efficient SQL queries, tuning database performance, and troubleshooting any related issues to improve the overall system's efficiency.
- AWS Aurora MySQL Management. Take charge of managing AWS Aurora MySQL, including writing stored procedures, setting up read replication, and ensuring database scalability. You will also be involved in troubleshooting performance issues and ensuring smooth database operations.
- Performance Monitoring and Troubleshooting. Proactively monitor AWS resources and systems to ensure optimal performance. You'll be responsible for identifying and resolving any issues that may hinder the performance of the data lake and associated services.
Required Skills & Experience
- 7-10+ years of experience in data engineering.
- Expertise in AWS data lake architecture, designing and implementing scalable data lakes.
- Proficiency with AWS Glue or DBT for ETL jobs and data cataloging.
- Strong experience with AWS RedShift or Snowflake for data warehousing and performance optimization.
- Solid skills in AWS SQS (Simple Queue Service) development for building event-driven architectures.
- Familiarity with AWS EventBridge Scheduler for automating scheduled tasks and workflows.
- Strong problem-solving and troubleshooting abilities to optimize the performance of data systems and architecture.
- Ability to work independently and collaborate with cross-functional teams to ensure successful data solutions.
Nice to Have Skills & Experience
- Experience with AWS data migration services for efficient data movement and transformation.
- Familiarity with the Hudi database for managing large datasets and ensuring data consistency.
- Experience in managing AWS Aurora MySQL databases, including performance tuning, replication setup, and stored procedure development.
- Proficiency in Python or Scala for scripting and automating data engineering tasks.
What You’ll Bring to the Role
- Innovative & Creative Mindset. You’ll be part of a team that thrives on creativity, constantly exploring new ways to enhance data architectures and processes to improve healthcare solutions.
- Expertise in AWS Technologies. With a strong background in AWS, you’ll bring valuable experience to optimize cloud infrastructure and services for maximum efficiency and performance.
- Collaborative Spirit. You’ll work closely with a global team, participating in brainstorming sessions and offering insights to improve data processes and systems.
- Leadership in Data Engineering. Your experience will guide the team in setting best practices for data lake management, ETL design, and performance optimization.
Why Join Insight Global?
This is an excellent opportunity to work with a dynamic, forward-thinking team that values innovation and creativity. As a Data Engineer with Insight Global, you'll have the chance to make a significant impact on a cutting-edge healthcare application, helping to improve outcomes through data-driven insights. You'll work with the latest AWS technologies and have the opportunity to showcase your expertise while collaborating with experts across various domains.
If you're ready to take your data engineering career to the next level, apply today to join Insight Global's team of talented professionals!