Aurora PostgreSQL and Iceberg: How to Query Lake Data Without ETL

Introduction

Applications often keep operational data in PostgreSQL while analytical data lives in a data lake. Moving information between these environments usually requires an ETL or ELT pipeline.

That architecture works, but it introduces another system to operate, another data copy to manage, and another step between data creation and analysis.

Amazon Aurora PostgreSQL now provides capabilities for working with Apache Iceberg data stored in Amazon S3. This creates an alternative architecture where PostgreSQL-based applications can access lake data without first copying that data into database tables.

The idea is particularly useful for applications that need to combine operational PostgreSQL data with large analytical datasets stored in an open table format.

This article explains how Aurora PostgreSQL and Iceberg can fit together, what the architecture looks like, how queries work, and what developers should consider before using it in production.

What Is Apache Iceberg?

Apache Iceberg is an open table format designed for large analytical datasets.

Instead of treating a collection of files in object storage as an unstructured folder, Iceberg maintains table metadata that describes:

Table schema
Data files
Table snapshots
Partition information
Schema evolution
Historical table states

A simplified architecture looks like this:

Application
     |
     v
Amazon S3
     |
     +---- Parquet / Data Files
     |
     +---- Iceberg Metadata

The data remains in object storage rather than being copied into a traditional relational table.

Iceberg is commonly used with analytical engines because it provides database-like table management over data stored in a data lake.

Why Connect Aurora PostgreSQL to Iceberg?

Aurora PostgreSQL is commonly used for transactional workloads.

For example:

Customers
Orders
Payments
Subscriptions
Products

An S3-based data lake may contain much larger datasets:

Historical Orders
Application Events
Clickstream Data
IoT Data
Logs
Analytics Data

Without an integration between the two environments, developers may need to create a pipeline:

Aurora PostgreSQL
       |
       v
ETL / ELT Pipeline
       |
       v
S3 Data Lake
       |
       v
Iceberg Tables

The pipeline introduces additional processing and operational dependencies.

A direct query architecture can instead look like:

Application / SQL Client
          |
          v
Aurora PostgreSQL
          |
          v
Iceberg Data in S3

This does not eliminate every form of data movement or processing. It changes where the query is executed and avoids requiring a traditional ETL pipeline simply to make lake data accessible to PostgreSQL workloads.

Aurora PostgreSQL and Data Lake Queries

The integration is useful when a PostgreSQL application needs access to data that already exists in an S3-based lake.

For example, suppose an organization has:

Aurora
  |
  +---- customers
  +---- orders

S3 + Iceberg
  |
  +---- five years of historical transactions
  +---- customer activity
  +---- application events

A reporting workflow may need to combine current operational information with historical data.

Instead of importing all historical records into Aurora, the application can query the relevant lake data through the supported integration.

The Basic Architecture

A simplified design is:

                +----------------------+
                |  Aurora PostgreSQL   |
                +----------+-----------+
                           |
                           |
                    SQL / Query Layer
                           |
                           v
                +----------------------+
                | Apache Iceberg Data |
                +----------+-----------+
                           |
                           v
                    Amazon S3
                           |
             +-------------+-------------+
             |                           |
        Parquet Data                Iceberg Metadata

The important distinction is that S3 remains the storage layer for the lake data.

Aurora provides a PostgreSQL-compatible environment from which the data can be accessed through the supported integration.

Why Avoid Traditional ETL?

ETL is not inherently bad.

It is useful when data needs to be transformed, cleaned, aggregated, or moved into a destination optimized for a particular workload.

The problem occurs when a pipeline exists only because the application cannot otherwise access the source data.

Consider this workflow:

S3
 |
 v
Extract
 |
 v
Transform
 |
 v
Load into Aurora
 |
 v
Query

If the dataset is large, the organization may need to manage:

Pipeline scheduling
Failure handling
Duplicate prevention
Data synchronization
Storage duplication
Schema changes
Operational monitoring

A query architecture can reduce some of these requirements when the analytical data can remain in S3.

Operational Data vs Lake Data

One important design decision is deciding which data belongs in Aurora and which belongs in the lake.

Aurora is well suited to transactional operations such as:

INSERT order
UPDATE customer
SELECT account
UPDATE payment status

Iceberg-based lake storage is better suited to large analytical datasets such as:

Analyze five years of transactions
Aggregate billions of events
Run historical reporting
Process large datasets

The architecture becomes:

Workload	Typical Location
Transaction processing	Aurora PostgreSQL
Current application state	Aurora PostgreSQL
Large historical datasets	S3 + Iceberg
Analytical data	S3 + Iceberg
High-volume event history	S3 + Iceberg
Operational queries	Aurora PostgreSQL

The exact boundary depends on workload characteristics.

A Practical Example

Imagine an e-commerce application.

Aurora contains:

customers
orders
products
payments

The data platform stores historical transaction information in an Iceberg table:

historical_orders

A business application may need to answer:

Which customers placed orders in the last five years,
and what was their historical purchase activity?

Instead of importing all historical records into Aurora, the application can query the lake data where appropriate.

Conceptually:

SELECT
    customer_id,
    COUNT(*) AS order_count,
    SUM(order_amount) AS total_spend
FROM historical_orders
WHERE order_date >= DATE '2021-01-01'
GROUP BY customer_id;

The exact SQL syntax and supported query behavior depend on the Aurora PostgreSQL version and the specific integration being used.

The architectural point is more important than the example: the historical dataset can remain in the lake rather than being copied into an operational database.

Why Iceberg Matters

The value is not simply that the data is stored in S3.

Iceberg provides table-level metadata and management capabilities on top of object storage.

A traditional folder of Parquet files can be difficult to manage as the dataset grows.

Iceberg adds concepts such as:

Table
 |
 +---- Schema
 |
 +---- Snapshot
 |
 +---- Data Files
 |
 +---- Metadata

This allows analytical systems to reason about the dataset as a table rather than as a collection of unrelated files.

Schema Evolution

Data structures change over time.

An order table might initially contain:

order_id
customer_id
amount

Later, the business may add:

currency
sales_channel
discount

Iceberg supports schema evolution capabilities that make these changes easier to manage than manually maintaining large collections of data files.

For application developers, this means the lake can evolve without necessarily requiring a complete rewrite of historical data.

However, consumers still need to understand which fields are guaranteed and which are optional.

Partitioning Matters

Large datasets should not be queried as though every row must be scanned for every request.

For example, a historical orders table could be organized around a date dimension:

historical_orders/
    year=2024/
    year=2025/
    year=2026/

Iceberg's metadata allows query engines to identify relevant data files more efficiently.

A query such as:

SELECT *
FROM historical_orders
WHERE order_date >= DATE '2026-01-01';

can benefit from appropriate partitioning and metadata.

Poor partition design can still result in unnecessary data processing.

Query Pushdown and Data Reduction

When querying large datasets, reducing the amount of data that needs to be processed is important.

Useful techniques include:

Selecting only required columns
Filtering early
Using appropriate partitioning
Avoiding unnecessary full-table scans
Aggregating at the appropriate level

Instead of:

SELECT *
FROM historical_orders;

prefer:

SELECT
    customer_id,
    SUM(order_amount) AS total_spend
FROM historical_orders
WHERE order_date >= DATE '2026-01-01'
GROUP BY customer_id;

The second query expresses a much more focused workload.

Aurora Is Not Automatically a Data Lake Engine

One important misconception should be avoided.

Connecting Aurora PostgreSQL to lake data does not mean Aurora suddenly becomes a replacement for every analytical engine.

Large-scale analytical workloads can have very different resource requirements from transactional workloads.

Before moving an analytical workload into an Aurora-based query architecture, evaluate:

Query complexity
Dataset size
Concurrent users
Query frequency
Processing requirements
Latency requirements
Cost
Impact on operational workloads

The architecture should protect transactional workloads from expensive analytical queries.

ETL vs Direct Lake Access

Area	Traditional ETL	Direct Lake Access
Data copy	Usually required	Can be avoided
Pipeline management	Required	Reduced
Data freshness	Pipeline-dependent	Can be closer to source
Storage duplication	Possible	Reduced
Transformation	Strong	Depends on query architecture
Operational complexity	Higher	Potentially lower
Large analytical workloads	Common	Workload-dependent
Data governance	Pipeline-based	Requires lake governance

Direct access does not eliminate data engineering.

It changes where some of the work happens.

Security Considerations

The integration also introduces a cross-service security boundary.

A production design should consider:

Aurora IAM permissions
S3 access policies
Encryption
Network configuration
Database roles
Data classification
Lake governance
Audit logging

Do not grant broad S3 permissions simply because the database needs to query a particular dataset.

Use narrowly scoped access to the required resources.

For sensitive datasets, access should be aligned with the same security policies used for other analytical systems.

Data Governance

A data lake can contain information from many applications.

That creates governance requirements around:

Data ownership
Retention
Classification
Personally identifiable information
Access control
Schema ownership
Data quality

For example:

Customer Data
      |
      +---- Owner: Customer Platform
      |
      +---- Classification: Sensitive
      |
      +---- Retention: Defined Policy
      |
      +---- Consumers: Approved Teams

Querying lake data directly does not remove these responsibilities.

Common Mistakes

Treating the Lake Like a Transactional Database

Large analytical scans can behave very differently from normal PostgreSQL queries.

Ignoring Data Layout

Poor partitioning and file organization can result in unnecessary data processing.

Copying Everything Into Aurora

If the goal is to avoid ETL, moving the entire lake into database tables defeats the architectural purpose.

Running Heavy Analytics Against Production

Separate analytical workloads from critical transaction processing when the workload could affect application performance.

Ignoring Schema Evolution

Lake schemas change over time. Consumers should be designed with those changes in mind.

Granting Broad Storage Permissions

Database access to S3 should follow least-privilege principles.

Troubleshooting

Query Cannot Access Lake Data

Check:

Aurora configuration
Required extensions or supported features
Database permissions
S3 permissions
Resource paths
Region configuration
Encryption and access policies

Query Is Too Slow

Investigate:

Data volume
Partitioning
File sizes
Filter selectivity
Columns being read
Query plan
Concurrent workloads

Results Do Not Match Expectations

Check:

Iceberg snapshot state
Schema versions
Data freshness
Partition configuration
Time-zone handling
Duplicate records

Production Workload Is Affected

Move heavy analytical queries away from critical transactional paths or introduce a separate architecture for analytical workloads.

Best Practices

Keep transactional data in Aurora when it requires transactional database behavior.
Keep large analytical datasets in S3 when a lake architecture is appropriate.
Use Iceberg for managed analytical table structures.
Design partitions around actual query patterns.
Select only the columns required by the application.
Filter large datasets early.
Avoid unnecessary data duplication.
Apply least-privilege access to S3 and database resources.
Monitor analytical workloads separately from transactions.
Define ownership for lake datasets.
Plan for schema evolution.
Test realistic data volumes before production deployment.
Use infrastructure as code for repeatable configuration.
Document which workloads should access the lake directly.

Advantages and Disadvantages

Advantages

Can reduce the need for traditional ETL pipelines
Allows lake data to remain in S3
Reduces unnecessary data duplication
Provides access to large historical datasets
Can simplify some application architectures
Works with open table formats such as Iceberg

Disadvantages

Not every analytical workload belongs in Aurora
Query performance depends heavily on data layout
Security configuration spans multiple AWS services
Data governance remains necessary
Large queries can affect operational workloads if poorly designed
Teams need to understand both PostgreSQL and lake-storage concepts

When Should You Use This Architecture?

Aurora PostgreSQL and Iceberg are a useful combination when an application needs SQL access to large datasets that already live in an S3-based data lake.

Good candidates include:

Historical reporting
Customer activity analysis
Transaction history
Operational applications that need selected analytical information
Applications combining current database state with historical lake data
Organizations trying to reduce unnecessary ETL pipelines

It is less appropriate when the workload requires intensive analytical processing better handled by a dedicated analytical engine.

Summary

Aurora PostgreSQL and Apache Iceberg provide a way to bring PostgreSQL applications closer to data stored in an S3-based lake without automatically copying that data into database tables.

The architecture separates responsibilities clearly: Aurora can continue handling transactional workloads while Iceberg and S3 hold large analytical datasets.

The biggest benefit is architectural flexibility. Teams can decide whether data should be queried where it already exists rather than creating another pipeline and another copy simply to make the data accessible.

For production systems, performance, security, data governance, partitioning, schema evolution, and workload isolation still matter. Direct lake access is not a replacement for every ETL or analytical architecture, but it can remove unnecessary data movement for the right workloads.