Introduction

Applications often keep operational data in PostgreSQL while analytical data lives in a data lake. Moving information between these environments usually requires an ETL or ELT pipeline.

That architecture works, but it introduces another system to operate, another data copy to manage, and another step between data creation and analysis.

Amazon Aurora PostgreSQL now provides capabilities for working with Apache Iceberg data stored in Amazon S3. This creates an alternative architecture where PostgreSQL-based applications can access lake data without first copying that data into database tables.

The idea is particularly useful for applications that need to combine operational PostgreSQL data with large analytical datasets stored in an open table format.

This article explains how Aurora PostgreSQL and Iceberg can fit together, what the architecture looks like, how queries work, and what developers should consider before using it in production.

What Is Apache Iceberg?

Apache Iceberg is an open table format designed for large analytical datasets.

Instead of treating a collection of files in object storage as an unstructured folder, Iceberg maintains table metadata that describes:

A simplified architecture looks like this:

Application
     |
     v
Amazon S3
     |
     +---- Parquet / Data Files
     |
     +---- Iceberg Metadata

The data remains in object storage rather than being copied into a traditional relational table.

Iceberg is commonly used with analytical engines because it provides database-like table management over data stored in a data lake.

Why Connect Aurora PostgreSQL to Iceberg?

Aurora PostgreSQL is commonly used for transactional workloads.

For example:

Customers
Orders
Payments
Subscriptions
Products

An S3-based data lake may contain much larger datasets:

Historical Orders
Application Events
Clickstream Data
IoT Data
Logs
Analytics Data

Without an integration between the two environments, developers may need to create a pipeline:

Aurora PostgreSQL
       |
       v
ETL / ELT Pipeline
       |
       v
S3 Data Lake
       |
       v
Iceberg Tables

The pipeline introduces additional processing and operational dependencies.

A direct query architecture can instead look like:

Application / SQL Client
          |
          v
Aurora PostgreSQL
          |
          v
Iceberg Data in S3

This does not eliminate every form of data movement or processing. It changes where the query is executed and avoids requiring a traditional ETL pipeline simply to make lake data accessible to PostgreSQL workloads.

Aurora PostgreSQL and Data Lake Queries

The integration is useful when a PostgreSQL application needs access to data that already exists in an S3-based lake.

For example, suppose an organization has:

Aurora
  |
  +---- customers
  +---- orders

S3 + Iceberg
  |
  +---- five years of historical transactions
  +---- customer activity
  +---- application events

A reporting workflow may need to combine current operational information with historical data.

Instead of importing all historical records into Aurora, the application can query the relevant lake data through the supported integration.

The Basic Architecture

A simplified design is:

                +----------------------+
                |  Aurora PostgreSQL   |
                +----------+-----------+
                           |
                           |
                    SQL / Query Layer
                           |
                           v
                +----------------------+
                | Apache Iceberg Data |
                +----------+-----------+
                           |
                           v
                    Amazon S3
                           |
             +-------------+-------------+
             |                           |
        Parquet Data                Iceberg Metadata

The important distinction is that S3 remains the storage layer for the lake data.

Aurora provides a PostgreSQL-compatible environment from which the data can be accessed through the supported integration.

Why Avoid Traditional ETL?

ETL is not inherently bad.

It is useful when data needs to be transformed, cleaned, aggregated, or moved into a destination optimized for a particular workload.

The problem occurs when a pipeline exists only because the application cannot otherwise access the source data.

Consider this workflow:

S3
 |
 v
Extract
 |
 v
Transform
 |
 v
Load into Aurora
 |
 v
Query

If the dataset is large, the organization may need to manage:

A query architecture can reduce some of these requirements when the analytical data can remain in S3.

Operational Data vs Lake Data

One important design decision is deciding which data belongs in Aurora and which belongs in the lake.

Aurora is well suited to transactional operations such as:

INSERT order
UPDATE customer
SELECT account
UPDATE payment status

Iceberg-based lake storage is better suited to large analytical datasets such as:

Analyze five years of transactions
Aggregate billions of events
Run historical reporting
Process large datasets

The architecture becomes:

Workload

Typical Location

Transaction processing

Aurora PostgreSQL

Current application state

Aurora PostgreSQL

Large historical datasets

S3 + Iceberg

Analytical data

S3 + Iceberg

High-volume event history

S3 + Iceberg

Operational queries

Aurora PostgreSQL

The exact boundary depends on workload characteristics.

A Practical Example

Imagine an e-commerce application.

Aurora contains:

customers
orders
products
payments

The data platform stores historical transaction information in an Iceberg table:

historical_orders

A business application may need to answer:

Which customers placed orders in the last five years,
and what was their historical purchase activity?

Instead of importing all historical records into Aurora, the application can query the lake data where appropriate.

Conceptually:

SELECT
    customer_id,
    COUNT(*) AS order_count,
    SUM(order_amount) AS total_spend
FROM historical_orders
WHERE order_date >= DATE '2021-01-01'
GROUP BY customer_id;

The exact SQL syntax and supported query behavior depend on the Aurora PostgreSQL version and the specific integration being used.

The architectural point is more important than the example: the historical dataset can remain in the lake rather than being copied into an operational database.

Why Iceberg Matters

The value is not simply that the data is stored in S3.

Iceberg provides table-level metadata and management capabilities on top of object storage.

A traditional folder of Parquet files can be difficult to manage as the dataset grows.

Iceberg adds concepts such as:

Table
 |
 +---- Schema
 |
 +---- Snapshot
 |
 +---- Data Files
 |
 +---- Metadata

This allows analytical systems to reason about the dataset as a table rather than as a collection of unrelated files.

Schema Evolution

Data structures change over time.

An order table might initially contain:

order_id
customer_id
amount

Later, the business may add:

currency
sales_channel
discount

Iceberg supports schema evolution capabilities that make these changes easier to manage than manually maintaining large collections of data files.

For application developers, this means the lake can evolve without necessarily requiring a complete rewrite of historical data.

However, consumers still need to understand which fields are guaranteed and which are optional.

Partitioning Matters

Large datasets should not be queried as though every row must be scanned for every request.

For example, a historical orders table could be organized around a date dimension:

historical_orders/
    year=2024/
    year=2025/
    year=2026/

Iceberg's metadata allows query engines to identify relevant data files more efficiently.

A query such as:

SELECT *
FROM historical_orders
WHERE order_date >= DATE '2026-01-01';

can benefit from appropriate partitioning and metadata.

Poor partition design can still result in unnecessary data processing.

Query Pushdown and Data Reduction

When querying large datasets, reducing the amount of data that needs to be processed is important.

Useful techniques include:

Instead of:

SELECT *
FROM historical_orders;

prefer:

SELECT
    customer_id,
    SUM(order_amount) AS total_spend
FROM historical_orders
WHERE order_date >= DATE '2026-01-01'
GROUP BY customer_id;

The second query expresses a much more focused workload.

Aurora Is Not Automatically a Data Lake Engine

One important misconception should be avoided.

Connecting Aurora PostgreSQL to lake data does not mean Aurora suddenly becomes a replacement for every analytical engine.

Large-scale analytical workloads can have very different resource requirements from transactional workloads.

Before moving an analytical workload into an Aurora-based query architecture, evaluate:

The architecture should protect transactional workloads from expensive analytical queries.

ETL vs Direct Lake Access

Area

Traditional ETL

Direct Lake Access

Data copy

Usually required

Can be avoided

Pipeline management

Required

Reduced

Data freshness

Pipeline-dependent

Can be closer to source

Storage duplication

Possible

Reduced

Transformation

Strong

Depends on query architecture

Operational complexity

Higher

Potentially lower

Large analytical workloads

Common

Workload-dependent

Data governance

Pipeline-based

Requires lake governance

Direct access does not eliminate data engineering.

It changes where some of the work happens.

Security Considerations

The integration also introduces a cross-service security boundary.

A production design should consider:

Do not grant broad S3 permissions simply because the database needs to query a particular dataset.

Use narrowly scoped access to the required resources.

For sensitive datasets, access should be aligned with the same security policies used for other analytical systems.

Data Governance

A data lake can contain information from many applications.

That creates governance requirements around:

For example:

Customer Data
      |
      +---- Owner: Customer Platform
      |
      +---- Classification: Sensitive
      |
      +---- Retention: Defined Policy
      |
      +---- Consumers: Approved Teams

Querying lake data directly does not remove these responsibilities.

Common Mistakes

Treating the Lake Like a Transactional Database

Large analytical scans can behave very differently from normal PostgreSQL queries.

Ignoring Data Layout

Poor partitioning and file organization can result in unnecessary data processing.

Copying Everything Into Aurora

If the goal is to avoid ETL, moving the entire lake into database tables defeats the architectural purpose.

Running Heavy Analytics Against Production

Separate analytical workloads from critical transaction processing when the workload could affect application performance.

Ignoring Schema Evolution

Lake schemas change over time. Consumers should be designed with those changes in mind.

Granting Broad Storage Permissions

Database access to S3 should follow least-privilege principles.

Troubleshooting

Query Cannot Access Lake Data

Check:

  1. Aurora configuration

  2. Required extensions or supported features

  3. Database permissions

  4. S3 permissions

  5. Resource paths

  6. Region configuration

  7. Encryption and access policies

Query Is Too Slow

Investigate:

Results Do Not Match Expectations

Check:

Production Workload Is Affected

Move heavy analytical queries away from critical transactional paths or introduce a separate architecture for analytical workloads.

Best Practices

  1. Keep transactional data in Aurora when it requires transactional database behavior.

  2. Keep large analytical datasets in S3 when a lake architecture is appropriate.

  3. Use Iceberg for managed analytical table structures.

  4. Design partitions around actual query patterns.

  5. Select only the columns required by the application.

  6. Filter large datasets early.

  7. Avoid unnecessary data duplication.

  8. Apply least-privilege access to S3 and database resources.

  9. Monitor analytical workloads separately from transactions.

  10. Define ownership for lake datasets.

  11. Plan for schema evolution.

  12. Test realistic data volumes before production deployment.

  13. Use infrastructure as code for repeatable configuration.

  14. Document which workloads should access the lake directly.

Advantages and Disadvantages

Advantages

Disadvantages

When Should You Use This Architecture?

Aurora PostgreSQL and Iceberg are a useful combination when an application needs SQL access to large datasets that already live in an S3-based data lake.

Good candidates include:

It is less appropriate when the workload requires intensive analytical processing better handled by a dedicated analytical engine.

Summary

Aurora PostgreSQL and Apache Iceberg provide a way to bring PostgreSQL applications closer to data stored in an S3-based lake without automatically copying that data into database tables.

The architecture separates responsibilities clearly: Aurora can continue handling transactional workloads while Iceberg and S3 hold large analytical datasets.

The biggest benefit is architectural flexibility. Teams can decide whether data should be queried where it already exists rather than creating another pipeline and another copy simply to make the data accessible.

For production systems, performance, security, data governance, partitioning, schema evolution, and workload isolation still matter. Direct lake access is not a replacement for every ETL or analytical architecture, but it can remove unnecessary data movement for the right workloads.