When working with large datasets, a Spark job that normally finishes in a few minutes can suddenly take much longer. Sometimes the job is not failing at all. It continues running, processes data, and eventually completes, but its execution time keeps increasing.
Finding the reason behind this behavior can be difficult, especially when an AWS Glue environment contains multiple ETL jobs, different data sources, and complex Spark transformations.
The problem is not always related to the amount of data. A slow Spark job can be caused by data skew, inefficient joins, excessive shuffling, memory pressure, insufficient parallelism, or poorly distributed workloads.
AWS Glue provides observability metrics that help developers identify these performance problems without manually investigating every job run. These metrics are available through Amazon CloudWatch and AWS Glue Studio, making it easier to understand where a Spark job spends its time and which resources are creating bottlenecks.
In this article, we will explore how AWS Glue Observability works, how to enable it, how to identify slow Spark jobs, and how to use the collected metrics to improve ETL performance.
Why Finding Slow Spark Jobs Is Difficult
Apache Spark processes data across multiple executors. A single ETL job can contain several stages, and each stage can execute multiple tasks in parallel.
For example, consider an ETL pipeline that:
- Reads customer records from Amazon S3.
- Cleans and transforms the records.
- Joins customer information with transaction data.
- Aggregates transaction amounts.
- Writes the final output to Amazon S3.
If the pipeline becomes slow, the problem could exist in any of these operations.
A join might create a large shuffle. One executor might process significantly more data than others. Another stage might spend time waiting for memory or disk resources.
Simply checking the total execution time does not reveal the actual problem.
Traditional monitoring usually tells developers whether a job succeeded, failed, or exceeded its expected execution time. However, understanding why it became slow requires more detailed information about the Spark workload.
This is where AWS Glue Observability metrics become useful.
What Is AWS Glue Observability?
AWS Glue Observability is a monitoring capability that generates additional metrics about AWS Glue for Apache Spark jobs.
These metrics are published to Amazon CloudWatch and help developers understand the internal behavior of their Spark applications.
Instead of relying only on job execution status, developers can investigate performance, resource utilization, data movement, and reliability.
AWS Glue Observability provides metrics across four important categories.
| Category | What it monitors | Why it matters |
|---|
| Reliability | Errors and failure patterns | Helps identify recurring job failures |
| Performance | Spark stage skewness | Helps identify slow tasks and uneven workloads |
| Throughput | Data read and write activity | Helps identify inefficient data movement |
| Resource utilization | Worker, memory, and disk usage | Helps identify resource bottlenecks |
These metrics are particularly useful when a job completes successfully but takes longer than expected.
AWS Glue Observability metrics are available for AWS Glue 4.0 and later versions.
Enabling AWS Glue Observability Metrics
Enabling observability is straightforward. Developers can configure it through AWS Glue Studio or the AWS CLI.
Enabling Observability Through AWS Glue Studio
Follow these steps:
- Open the AWS Glue console.
- Navigate to ETL jobs.
- Select the Spark job you want to monitor.
- Open the Job details tab.
- Expand Advanced properties.
- Locate Job observability metrics.
- Enable the observability metrics option.
- Save the job configuration.
After enabling this option, AWS Glue publishes additional observability metrics to CloudWatch when the job runs.
Enabling Observability Through AWS CLI
If you manage AWS Glue jobs through scripts or infrastructure automation, you can enable observability using a job argument.
Add the following argument to the job configuration:
--enable-observability-metrics true
For example, when starting a job using AWS CLI:
aws glue start-job-run \
--job-name CustomerDataETL \
--arguments '{"--enable-observability-metrics":"true"}'
This configuration enables observability for the job run without requiring developers to modify the Spark transformation logic.
For consistent monitoring, configure the argument in the job definition or your deployment configuration rather than relying on developers to remember it for individual executions.
One important detail is that observability metrics and standard job metrics are separate monitoring settings. Make sure the required metrics are enabled for the workload you want to investigate.
Finding Slow Jobs Through CloudWatch Metrics
Once observability metrics are enabled, the next step is to examine them.
AWS Glue publishes observability metrics to Amazon CloudWatch approximately every 30 seconds. This allows developers to monitor how a job behaves during execution instead of waiting until it finishes.
To access these metrics:
- Open the Amazon CloudWatch console.
- Navigate to All metrics.
- Select the AWS Glue namespace.
- Open Job Observability Metrics.
- Select the job name and job run ID.
- Choose the metrics you want to investigate.
You can also explore Observability Metrics Per Source and Observability Metrics Per Sink to understand individual data sources and destinations.
The most useful metrics depend on the type of performance problem you are investigating.
Identifying Data Skew
Data skew is one of the common reasons behind slow Spark jobs.
It occurs when data is distributed unevenly across partitions. Some executors finish their tasks quickly, while others continue processing much larger partitions.
For example, imagine a transaction dataset containing millions of records. If one customer account has significantly more transactions than other accounts, a group-by operation or join involving that account can create an uneven workload.
AWS Glue provides the following metrics:
glue.driver.skewness.stage
glue.driver.skewness.job
The stage-level metric helps identify skewness within individual Spark stages.
The job-level metric represents the maximum weighted skewness across stages, taking stage duration into account.
When these metrics indicate skewness, investigate the affected stage rather than immediately increasing the worker count.
Possible solutions include:
- Reviewing partition distribution.
- Repartitioning data using an appropriate key.
- Using broadcast joins when one dataset is sufficiently small.
- Enabling Spark Adaptive Query Execution where supported.
- Reviewing skewed join behavior and Spark configuration.
The objective is to distribute processing work more evenly across executors.
Identifying Low Worker Utilization
Adding more workers does not automatically make a Spark job faster.
Sometimes a job has more allocated capacity than its workload can effectively use.
AWS Glue provides the following metric:
glue.driver.workerUtilization
This metric indicates the percentage of allocated workers that are actually being used.
For example, if a job has several workers allocated but consistently uses only a small portion of the available capacity, the job may not benefit from its current configuration.
Possible reasons include:
- Insufficient parallel tasks.
- A small input dataset.
- A bottleneck in one processing stage.
- Operations that cannot execute efficiently in parallel.
AWS Glue Auto Scaling can help adjust worker capacity based on workload requirements.
However, before enabling it, investigate whether the job has enough parallel work to use additional workers effectively.
Monitoring Memory and Disk Usage
Memory pressure can also increase Spark execution time.
When executors experience high memory utilization, Spark may spend additional time managing memory, spilling intermediate data to disk, or processing large partitions.
AWS Glue provides metrics such as:
glue.driver.memory.heap.used.percentage
glue.ALL.memory.heap.used.percentage
glue.driver.disk.used.percentage
glue.ALL.disk.used.percentage
These metrics help developers understand memory and disk utilization at the driver and executor levels.
For example, consistently high executor memory usage might indicate that a transformation is processing large partitions or retaining unnecessary intermediate data.
Possible improvements include:
- Reducing unnecessary columns before joins.
- Avoiding unnecessary caching.
- Reviewing large group-by operations.
- Adjusting partition sizes.
- Selecting an appropriate worker configuration.
Do not increase memory or worker capacity without first understanding which operation is consuming the resources.
Monitoring Data Throughput
Sometimes the Spark transformation logic is not the main problem. The job may spend most of its time reading data from a source or writing results to a destination.
AWS Glue provides throughput metrics that help developers monitor data movement.
For example:
glue.driver.bytesRead
glue.driver.recordsRead
glue.driver.filesRead
These metrics help identify changes in input data volume.
AWS Glue also provides source and sink observability metrics, allowing developers to investigate individual data movement operations.
Consider an ETL job that normally processes a few hundred files but suddenly receives thousands of small files.
Even if the total data volume remains similar, file listing, metadata operations, and task scheduling can introduce additional overhead.
By comparing throughput metrics across job runs, developers can investigate whether a change in input volume or file distribution is contributing to the slowdown.
Using Spark UI to Investigate the Actual Bottleneck
CloudWatch metrics provide a useful overview, but sometimes developers need more detailed information about the Spark execution plan.
This is where the Apache Spark UI becomes important.
AWS Glue provides a Spark UI that helps developers inspect:
- Spark jobs and stages.
- Task execution duration.
- Shuffle read and write operations.
- Executor activity.
- Logical and physical query plans.
For example, if observability metrics indicate stage skewness, open the Spark UI and investigate the affected stage.
Sort the jobs by execution duration and examine the stages that consume the most time.
A stage with a small number of active tasks but a long execution duration might indicate insufficient parallelism or an operation that cannot be distributed effectively.
Similarly, large shuffle read and write values can indicate expensive joins, aggregations, or repartitioning operations.
CloudWatch helps identify where to investigate, while Spark UI provides deeper information about why the workload is slow.
Using AWS Glue Job Run Insights for Failure Investigation
AWS Glue also provides a feature called Job Run Insights.
However, it is important to understand the difference between Job Run Insights and Observability metrics.
Observability metrics help investigate performance behavior, including slow but successful jobs.
Job Run Insights focuses on failed job executions. It can identify the script line associated with a failure, the last Spark action executed before the failure, relevant exception events, and possible root causes with recommended actions.
You can enable it using the following job argument:
--enable-job-insights true
When an eligible job run fails, AWS Glue can generate additional log streams in CloudWatch containing exception analysis and rule-based recommendations.
Successful job runs do not generate these failure-analysis streams.
Using both features gives developers a more complete monitoring approach: Observability metrics for performance investigation and Job Run Insights for troubleshooting failures.
Best Practices for Monitoring AWS Glue Jobs
Establish a Performance Baseline
Before optimizing a Spark job, understand its normal behavior.
Record execution duration, input volume, worker utilization, memory consumption, and shuffle activity.
Compare new job runs against this baseline to identify unusual changes.
Monitor Trends Instead of Individual Metrics
A single metric rarely explains the complete problem.
For example, high memory utilization does not automatically mean that memory is the only bottleneck.
Compare memory usage with shuffle activity, stage duration, and worker utilization to understand the actual workload.
Configure CloudWatch Alarms
Create alarms for important conditions such as unexpected execution duration, recurring failures, or abnormal resource utilization.
This helps teams identify problems before they affect downstream data pipelines.
Optimize Only After Identifying the Bottleneck
Avoid changing worker types, increasing worker counts, or modifying Spark configurations without understanding the underlying problem.
Measure the impact of every optimization and compare the results with the original baseline.
Keep Monitoring Consistent Across Environments
Use consistent observability settings across development, testing, and production environments.
This makes it easier to compare job behavior and investigate performance differences after deployment.
Advantages and Disadvantages of AWS Glue Observability
| Advantages | Disadvantages |
|---|
| Provides detailed performance metrics | Requires configuration and monitoring setup |
| Integrates with Amazon CloudWatch | Does not automatically fix slow jobs |
| Helps identify data skewness | Metrics alone may not explain every bottleneck |
| Monitors worker and resource utilization | Detailed investigation may still require Spark UI |
| Supports monitoring across multiple job runs | Requires developers to understand Spark metrics |
| Helps identify throughput changes | Metrics must be interpreted in the context of workload changes |
Common Mistakes Developers Should Avoid
Some common mistakes can make performance investigations less effective.
| Mistake | Why it causes problems | Better approach |
|---|
| Increasing workers immediately | May increase costs without improving execution time | Investigate worker utilization first |
| Ignoring data skewness | A few slow tasks can delay an entire stage | Examine stage-level skewness |
| Monitoring only total execution time | Does not identify the underlying bottleneck | Monitor individual metrics |
| Ignoring shuffle operations | Expensive data movement can slow transformations | Inspect Spark UI shuffle metrics |
| Treating Job Run Insights as performance monitoring | It focuses on failed executions | Use Observability metrics for performance |
| Ignoring input data changes | More files or uneven data can affect execution | Compare throughput across runs |
Troubleshooting Common Problems
Observability Metrics Are Missing
First, verify that the job uses AWS Glue 4.0 or later.
Check whether Job observability metrics is enabled in the job configuration. Also verify that the job has executed and that you are selecting the correct CloudWatch namespace, job name, and job run ID.
Worker Utilization Is Consistently Low
Check the Spark UI to understand how many tasks are running simultaneously.
The job might have limited parallelism or a processing stage that cannot effectively use all allocated workers.
Review the partition count and transformation logic before changing the worker configuration.
Job Skewness Is High
Identify the affected Spark stage and inspect its tasks.
Look for uneven partition sizes, large joins, and operations involving highly concentrated keys.
Consider repartitioning or using Adaptive Query Execution where appropriate.
Job Duration Increases After a Data Update
Compare the latest run with previous executions.
Check whether the number of input files, records, or bytes has increased. Also examine whether the new data distribution has introduced skewness.
The problem may be related to the input dataset rather than a change in the Spark script.
Summary
Slow Spark jobs are not always easy to troubleshoot, especially when an AWS Glue job completes successfully but takes longer than expected.
AWS Glue Observability provides additional visibility into performance, reliability, throughput, and resource utilization. By using these metrics with Amazon CloudWatch, developers can identify potential bottlenecks such as data skewness, low worker utilization, memory pressure, and inefficient data movement.
For deeper investigation, the Apache Spark UI helps identify expensive stages, slow tasks, and shuffle operations. AWS Glue Job Run Insights complements these capabilities by providing additional information when a job fails.
The most effective approach is to establish a performance baseline, monitor important metrics, investigate the actual bottleneck, and make targeted changes.
Instead of increasing worker capacity whenever a Spark job becomes slow, developers can use AWS Glue observability to understand what is happening inside the workload and make better optimization decisions.
Join the conversation! Your thoughts help the community grow.