Automatically Compacting Apache Iceberg Tables With AWS Glue

Apache Iceberg is widely used for building data lakes on Amazon S3 because it supports large analytical datasets, schema evolution, time travel, and reliable table updates. However, as data continues to arrive, Iceberg tables can accumulate thousands of small files.

This usually happens when applications write data in frequent batches, streaming pipelines continuously update tables, or multiple jobs produce small output files.

Although these files contain valid data, having too many of them can affect query performance. Analytical engines must spend additional time opening files, reading metadata, and planning queries.

AWS Glue Data Catalog provides managed table optimization for Apache Iceberg. One of its important capabilities is automatic compaction, which combines smaller data files into larger files without requiring developers to maintain separate compaction jobs.

In this article, we will explore why Iceberg tables need compaction, how AWS Glue manages the process, how to enable automatic compaction using the AWS CLI, and how to monitor optimization activities in a production data lake.

Understanding Small File Problems in Apache Iceberg

Apache Iceberg organizes analytical data into data files stored in locations such as Amazon S3.

Each table contains metadata that tracks its snapshots, manifests, and data files.

When a data pipeline writes small batches frequently, it can generate many small Parquet files instead of a smaller number of efficiently sized files.

For example, imagine a streaming application that processes customer transactions every few minutes.

A simplified storage layout might look like this:

s3://company-data/transactions/

    data/
        file-001.parquet
        file-002.parquet
        file-003.parquet
        file-004.parquet
        file-005.parquet
        file-006.parquet
        file-007.parquet
        file-008.parquet

Suppose each file contains only a small amount of data.

When an analytical query reads this table, the query engine must identify the relevant files and process their metadata.

As the number of files increases, query planning and file management can become less efficient.

Common problems include:

Increased metadata management overhead.
More S3 object requests.
Longer query planning time.
Inefficient analytical reads.
Increased operational complexity.

The solution is data file compaction.

What Is Apache Iceberg Compaction?

Compaction is a maintenance operation that combines multiple small data files into fewer, larger files.

For example, eight small files can be rewritten into two larger files.

Before compaction:

file-001.parquet
file-002.parquet
file-003.parquet
file-004.parquet
file-005.parquet
file-006.parquet
file-007.parquet
file-008.parquet

After compaction:

compacted-001.parquet
compacted-002.parquet

The Iceberg table metadata is updated to reference the rewritten files.

Compaction does not change the logical records stored in the table. It changes how those records are physically organized.

Apache Iceberg supports different compaction strategies, including:

Binpack

Binpack combines smaller files into larger files without intentionally sorting the records.

It is generally the preferred option for standard file-size optimization.

Sort

Sort reorganizes records according to specified columns.

For example, a transaction table frequently queried by transaction date can benefit from organizing records around that column.

This can improve data skipping and reduce unnecessary file reads.

Z-order

Z-order organizes records using multiple columns to improve data locality across dimensions.

It can be useful when analytical queries frequently filter on several columns together.

AWS Glue supports these strategies for its managed Iceberg compaction, subject to the supported Iceberg version and configuration requirements.

How AWS Glue Automatic Compaction Works

AWS Glue Data Catalog provides a managed table optimizer that monitors Iceberg tables and performs compaction when configured thresholds are reached.

The process works without requiring developers to create and schedule a separate Apache Spark maintenance job.

A simplified workflow looks like this:

Data Ingestion
      |
      v
Apache Iceberg Table
      |
      v
Amazon S3 Data Files
      |
      v
AWS Glue Table Optimizer
      |
      v
Identify Small Files
      |
      v
Rewrite Data Files
      |
      v
Update Iceberg Metadata
      |
      v
Optimized Table

AWS Glue monitors the table and its partitions.

According to its managed compaction configuration, the optimizer considers the number of files and their sizes. The documented default trigger is more than 100 files, with each file smaller than 75% of the configured target file size.

The write.target-file-size-bytes table property controls the target size and defaults to 512 MB when it is not specified.

Managed compaction currently supports Parquet-format Iceberg tables.

One important benefit is that compaction uses Iceberg's transactional metadata operations. Concurrent queries can continue accessing the table while the optimization takes place.

Step 1: Prepare the Iceberg Table

Before enabling automatic compaction, make sure the table is registered in the AWS Glue Data Catalog.

For example, an Iceberg table might contain transaction records with the following structure:

CREATE TABLE iceberg_db.transactions (
    transaction_id BIGINT,
    customer_id STRING,
    transaction_date DATE,
    amount DECIMAL(12, 2),
    status STRING
)
USING iceberg
LOCATION 's3://company-data/transactions/'
TBLPROPERTIES (
    'format-version' = '2',
    'write.format.default' = 'parquet',
    'write.target-file-size-bytes' = '536870912'
);

This example uses Apache Spark SQL with an appropriately configured Iceberg catalog.

The important properties are:

format-version: Defines the Iceberg table format version.
write.format.default: Specifies Parquet as the data file format.
write.target-file-size-bytes: Sets the target data file size to 512 MB.

The table location should be dedicated to this Iceberg table. Avoid sharing its storage location with unrelated tables.

Step 2: Create an IAM Role for Compaction

AWS Glue needs permission to read data files, write optimized files, and update Iceberg metadata.

Create an IAM role that AWS Glue can assume.

Its trust policy should allow the Glue service:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "glue.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

The role also needs appropriate permissions for:

Reading and writing objects in the table's S3 location.
Reading and updating Glue Data Catalog metadata.
Writing optimizer execution logs to CloudWatch.
Accessing Lake Formation data when applicable.
Decrypting and generating data keys when using supported SSE-KMS encryption.

The principal enabling the optimizer also needs permission to pass the IAM role to AWS Glue.

Use resource-specific IAM policies instead of granting unrestricted access to every S3 bucket or Data Catalog table.

Step 3: Enable Automatic Compaction Using AWS CLI

AWS Glue provides the create-table-optimizer command to enable compaction for an individual Iceberg table.

Run the following command:

aws glue create-table-optimizer \
  --catalog-id 123456789012 \
  --database-name iceberg_db \
  --table-name transactions \
  --type compaction \
  --table-optimizer-configuration '{
    "roleArn": "arn:aws:iam::123456789012:role/iceberg-optimizer-role",
    "enabled": true,
    "compactionConfiguration": {
      "icebergConfiguration": {
        "strategy": "binpack",
        "minInputFiles": 5
      }
    }
  }'

Replace the account ID, database, table, and IAM role with your actual values.

The configuration contains several important properties.

roleArn identifies the IAM role used by AWS Glue to perform optimization.
enabled activates the optimizer.
strategy specifies the compaction approach. In this example, Binpack is selected.
minInputFiles specifies the minimum number of input files required for compaction.

The configured minimum input file count does not override the service's other eligibility conditions.

For most standard data lake workloads, Binpack is a sensible starting strategy.

If the table requires sorting or Z-order optimization, evaluate those strategies separately based on the supported Iceberg version and query patterns.

Step 4: Verify the Compaction Configuration

After creating the optimizer, retrieve its configuration:

aws glue get-table-optimizer \
  --catalog-id 123456789012 \
  --database-name iceberg_db \
  --table-name transactions \
  --type compaction

Review the response to confirm that:

The optimizer is enabled.
The expected IAM role is configured.
The selected compaction strategy is correct.
The table optimizer configuration is valid.

You can also use the AWS Glue console.

Navigate to Data Catalog, select Tables, open the Iceberg table, and inspect the Table optimization tab.

The console provides visibility into the optimizer configuration and execution history.

Step 5: Monitor Compaction Runs

Enabling automatic compaction does not mean that every table is immediately rewritten.

AWS Glue monitors the table and starts optimization when its eligibility conditions are satisfied.

To inspect optimization history, use:

aws glue list-table-optimizer-runs \
  --catalog-id 123456789012 \
  --database-name iceberg_db \
  --table-name transactions \
  --type compaction

Review the execution information to identify successful and failed optimization runs.

Useful monitoring information includes:

Run status.
Start time.
Execution duration.
Number of files compacted.
Number of bytes compacted.
Failure details.

CloudWatch metrics can also help track compaction activity.

Two useful metrics are:

numberOfFilesCompacted
numberOfBytesCompacted

These metrics help determine whether the optimizer is reducing file fragmentation over time.

Compaction vs. Snapshot Retention vs. Orphan File Deletion

AWS Glue provides three different Iceberg table optimizers.

They solve different storage and maintenance problems.

Optimizer	Purpose	Main benefit
Compaction	Combines small data files	Improves data file layout
Snapshot retention	Removes expired snapshots	Controls historical metadata growth
Orphan file deletion	Removes unreferenced files	Reclaims unnecessary storage

These operations should not be treated as interchangeable.

Compaction rewrites data files.

Snapshot retention removes old table snapshots according to retention rules.

Orphan file deletion identifies files that are no longer referenced by Iceberg metadata.

For a production data lake, these optimizers can complement each other, but their retention settings must reflect business and recovery requirements.

Common Problems During Automatic Compaction

Compaction is not starting

The table might not meet the file-count or file-size eligibility conditions.

Check the number of files in the table and its partitions, the target file size, and the configured minimum input file count.

Also verify that the table uses the supported Parquet format.

IAM permission errors

The optimizer role might lack permission to access S3, update Data Catalog metadata, or write CloudWatch logs.

Review the IAM role policy and trust relationship.

If Lake Formation manages the table permissions, verify the required Lake Formation grants.

Compaction fails repeatedly

Concurrent writes can create optimistic concurrency conflicts.

Review the optimizer execution logs and configure appropriate retry behavior for the workload.

AWS Glue automatically suspends an optimizer after four consecutive compaction failures. Investigate the underlying problem and re-enable the optimizer after resolving it.

Compaction does not reduce storage usage

Compaction primarily improves data file organization. It does not automatically remove every historical snapshot or orphan file.

Review snapshot retention and orphan file deletion separately.

Best Practices for Production Iceberg Tables

Choose the target file size based on query patterns and data volume. A larger file is not automatically better for every workload.
Use Binpack as the initial strategy for standard compaction requirements.
Monitor partition-level file counts instead of looking only at the total table size.
Configure an IAM role with minimum required permissions.
Review concurrent write behavior when streaming applications update the same table.
Keep snapshot retention policies aligned with time travel and recovery requirements.
Configure orphan file retention carefully to avoid deleting files involved in delayed commits.
Avoid applying S3 lifecycle expiration rules to active Iceberg data and metadata paths.
Keep each optimized table's S3 location separate from unrelated datasets.
Monitor optimizer execution history and CloudWatch metrics regularly.
Test compaction on representative datasets before enabling it across a large data lake.
Review optimizer costs and actual query improvements to validate the operational benefit.

Advantages and Disadvantages of AWS Glue Managed Compaction

Advantages

Reduces the number of small files in Iceberg tables.
Improves data organization for analytical workloads.
Removes the need to maintain a separate scheduled compaction job.
Supports automatic optimization through the Data Catalog.
Integrates with AWS analytics services such as Athena and EMR.
Provides execution details and monitoring metrics.

Disadvantages

Requires additional configuration and IAM permissions.
Managed compaction introduces optimization costs.
Only supported table formats and configurations can be optimized.
Concurrent updates can cause compaction conflicts.
Automatic compaction does not replace snapshot retention or orphan file cleanup.
Poorly selected optimization strategies may not improve the application's actual query patterns.

Summary

Apache Iceberg tables can accumulate thousands of small files when applications continuously write data in small batches. This increases metadata overhead and can affect analytical query performance.

AWS Glue Data Catalog provides managed compaction that automatically combines smaller Parquet files into larger files when configured eligibility conditions are met.

Developers can enable the optimizer through the AWS Glue console or AWS CLI, configure an appropriate IAM role, select a compaction strategy, and monitor execution through optimizer history and CloudWatch metrics.

For production data lakes, compaction should be part of a broader table maintenance strategy that also considers snapshot retention, orphan file deletion, concurrent writes, and recovery requirements.

Automating these operations helps maintain a more efficient Iceberg storage layout without requiring teams to manage every maintenance task through custom ETL jobs.