S&P Global Commodity Insights is seeking a skilled Senior Data Engineer to join its dynamic Platform Solutions group. As part of the wider Commodity Insights team, you will work with a global group of professionals across two business lines, Fuel, Chemicals, and Resource Solutions and Gas, Power, and Climate Solutions. The role involves handling complex data tasks, building scalable data pipelines, and optimizing data processes to deliver actionable insights to key stakeholders. This is a fantastic opportunity for individuals who are passionate about data engineering and looking to make an impact in a fast-paced, high-growth environment.
What You'll Be Doing
- Data Munging & Analytics. You will combine diverse datasets using Python and large-scale data analytics tools, ensuring efficient transformation and integration across different systems. Your efforts will drive the company’s data infrastructure and contribute to the success of data-driven projects.
- Root Cause Analysis & Debugging. If data outputs do not meet expectations, you’ll be the go-to person to perform root cause analysis, identify bugs in the code, and work on optimizing data solutions.
- Data Transformation Pipeline Management. A significant part of your responsibility will be maintaining and improving existing data transformation pipelines that are built using Python and Spark. Your experience will ensure that these pipelines remain efficient and scalable as the company grows.
- Data Infrastructure & Cloud Technologies. You will build and enhance infrastructure for the optimal extraction, transformation, and loading of data. This will involve using a mix of SQL, Python, and AWS tools such as AWS S3, AWS RDS, and AWS ECS to handle large-scale data systems.
- Collaboration with Stakeholders. Working closely with product managers, data scientists, and design teams, you will address data-related technical issues and ensure that data infrastructure supports the needs of various stakeholders.
- Tool Development for Analytics. You’ll develop custom tools to assist analysts and data scientists in building and optimizing the company’s products, contributing to S&P Global’s position as an industry leader.
- Project & Stakeholder Management. You will take responsibility for managing projects, ensuring that deliverables are of high quality and are completed on time. You’ll also engage with various stakeholders to ensure that project requirements are met, and any data challenges are resolved efficiently.
- Industry Knowledge. You will deepen your understanding of the maritime industry and its connection to the global financial markets and shipping industry, helping the company tailor its products to clients in these sectors.
Basic Qualifications:
- Experience. A total of 5-8 years in data engineering roles, with at least 3 years of hands-on experience in large-scale data analytics and Python.
- Technical Expertise. Expertise in Python (including PySpark and Pandas) and experience working with databases like Aurora PostgreSQL RDS and SQLServer. Strong command of AWS tools such as S3, RDS, and ECS.
- Data Engineering Skills. Proven ability to manipulate, process, and extract value from large and complex datasets, ensuring high-quality results.
- Process Development. Hands-on experience building processes for data transformation, structuring data, and managing dependencies.
- Collaboration & Communication. Strong ability to work with cross-functional teams, including data engineers, scientists, and business stakeholders, while addressing data-related technical issues.
- Education. A Bachelor’s or higher degree in a quantitative field such as Computer Science, Statistics, Informatics, or a similar discipline from a reputable institution.
Must-Have Technical Skills
- Python (PySpark, Pandas)
- Aurora PostgreSQL RDS and SQLServer
- AWS Services (S3, RDS, ECS)
- Databricks
- General Programming Concepts & Design Patterns
Nice to Have
- Experience with Jenkins (CI/CD DevOps)
- Familiarity with AWS CloudFormation or Terraform
- Understanding of the ELK stack and NoSQL (MongoDB, Redis, Elasticsearch)
- Experience with AWS Lambda, Redshift, or other AWS services
- Knowledge of additional languages like Java, .NET, or Shell Scripting
Beneficial Skills
- Experience leading complex analytical projects and managing multiple stakeholders.
- Familiarity with geo-spatial data and unstructured data.
- Experience using big data tools (BigQuery, Hive, MongoDB, Cassandra, etc.) for data manipulation and modeling.
About S&P Global
S&P Global is a leading provider of essential intelligence that empowers organizations, governments, and individuals to make informed decisions. With over 35,000 employees globally, the company offers unmatched expertise and advanced technologies to help businesses navigate the evolving economic landscape. S&P Global's Commodity Insights division plays a pivotal role in enabling organizations to understand global energy and commodities markets, providing data-driven solutions that lead to progress and help clients build long-term, sustainable value.
Why Join Us?
At S&P Global, you’ll work with talented professionals from around the world, contributing to high-impact projects that shape the future of the energy, commodities, and financial markets. The company fosters a collaborative, inclusive culture where diverse perspectives are valued. Whether you’re interested in continuous learning, career growth, or making a tangible impact on the global economy, S&P Global offers opportunities for professional advancement and personal fulfillment.