Aurora PostgreSQL and Parquet: How to Query Data Without ETL

Introduction

Parquet has become a common storage format for analytical data because it stores data in a column-oriented structure and works well with large datasets in object storage.

Many applications, however, still use PostgreSQL for operational workloads. This creates a common architecture problem: application data lives in PostgreSQL while historical or analytical data is stored as Parquet files in Amazon S3.

The traditional solution is to build an ETL or ELT pipeline that copies and transforms the Parquet data before it can be queried from a database-oriented application.

Aurora PostgreSQL provides capabilities that can make this architecture simpler by allowing PostgreSQL workloads to work with data stored externally in S3, including Parquet-based datasets.

The result can be a design where the data remains in its original analytical storage location rather than being copied into Aurora simply to make it queryable.

What Is Parquet?

Apache Parquet is a columnar file format designed for efficient analytical workloads.

A traditional row-oriented format stores records approximately like this:

Row 1: customer_id, order_date, amount, status
Row 2: customer_id, order_date, amount, status
Row 3: customer_id, order_date, amount, status

Parquet organizes data by columns:

customer_id
customer_id
customer_id

order_date
order_date
order_date

amount
amount
amount

status
status
status

This layout can be efficient when a query needs only a few columns from a large dataset.

For example:

SELECT customer_id, amount
FROM historical_orders
WHERE order_date >= DATE '2026-01-01';

A column-oriented format can avoid processing unrelated columns when the underlying query engine supports the appropriate column and predicate pruning.

Why Store Parquet in Amazon S3?

Amazon S3 is commonly used as the storage layer for data lakes.

A simplified structure might look like:

S3
 |
 +---- orders/
 |      |
 |      +---- 2024/
 |      +---- 2025/
 |      +---- 2026/
 |
 +---- customers/
 |
 +---- events/

The actual files may be Parquet:

orders-0001.parquet
orders-0002.parquet
orders-0003.parquet

This approach separates storage from database compute.

Large historical datasets can remain in S3 while transactional data continues to live in Aurora PostgreSQL.

Why Query Parquet From Aurora PostgreSQL?

Consider an application with this architecture:

Aurora PostgreSQL
     |
     +---- customers
     +---- active_orders
     +---- payments

S3
     |
     +---- historical_orders.parquet
     +---- application_events.parquet

Suppose a reporting feature needs five years of historical order information.

One approach is:

Parquet
   |
   v
ETL
   |
   v
Aurora Table
   |
   v
Application Query

That introduces another copy of the data.

With an external-data architecture, the application can access the data where it already exists:

Application
    |
    v
Aurora PostgreSQL
    |
    v
Parquet in S3

This can reduce unnecessary data movement for suitable workloads.

The Architecture

A simplified design looks like this:

                  Application
                       |
                       v
              Aurora PostgreSQL
                       |
                  SQL Query
                       |
                       v
                 Parquet Data
                       |
                       v
                  Amazon S3

The database and object storage serve different purposes.

Aurora can remain responsible for transactional application data, while S3 stores larger analytical datasets.

A Practical Example

Imagine an e-commerce application.

Aurora contains current operational information:

customers
orders
products
payments

S3 contains historical transactions:

historical_orders/
    2024/
    2025/
    2026/

The application wants to calculate historical sales for a customer.

Instead of importing all historical records into Aurora, the relevant Parquet dataset can remain in S3.

A conceptual query could look like:

SELECT
    customer_id,
    SUM(order_amount) AS total_spend
FROM historical_orders
WHERE order_date >= DATE '2025-01-01'
GROUP BY customer_id;

The exact syntax depends on the Aurora PostgreSQL capability and integration being used.

The architectural idea is that the Parquet data does not have to become a traditional Aurora table simply because an application needs to query it.

Why Parquet Works Well for Analytical Data

Parquet provides several characteristics that make it attractive for lake storage.

Columnar Storage

Analytical queries frequently access a subset of columns.

For example:

SELECT customer_id, order_amount
FROM historical_orders;

A columnar format is well suited to this type of access.

Compression

Parquet supports compression, reducing storage requirements and potentially reducing the amount of data that needs to be processed.

Schema Information

Parquet files contain schema information describing their columns and data types.

Ecosystem Compatibility

Parquet is supported by many data-processing and analytics systems, making it useful when multiple platforms need to access the same data.

Parquet vs Iceberg

Parquet and Iceberg solve different problems.

Parquet is primarily a file format.

Iceberg is a table format that manages collections of data files and associated metadata.

Feature	Parquet	Iceberg
Type	File format	Table format
Columnar storage	Yes	Usually uses formats such as Parquet
Schema	File-level	Table-level
Table snapshots	No	Yes
Schema evolution	Basic file schema	Advanced table-level management
Partition management	File/layout dependent	Table-level
Historical snapshots	No	Yes
Best use	Individual analytical files	Managed analytical tables

This distinction matters when designing a data lake.

A collection of Parquet files may be enough for simple workloads.

Iceberg becomes more useful when the dataset needs table-level management, snapshots, schema evolution, and other data-lake capabilities.

Querying Only the Data You Need

Large Parquet datasets should not be treated like small database tables.

A query such as:

SELECT *
FROM historical_orders;

can require processing a significant amount of data.

Instead, select only the necessary columns:

SELECT
    customer_id,
    order_date,
    order_amount
FROM historical_orders
WHERE order_date >= DATE '2026-01-01';

Then apply selective filters where possible.

This is important for both performance and cost.

Partitioning Parquet Data

Large datasets should generally be organized around common query patterns.

For example:

historical_orders/
    year=2024/
    year=2025/
    year=2026/

A query filtering on the date may then avoid processing irrelevant partitions when the underlying query architecture supports partition pruning.

Another example could be:

events/
    region=us/
    region=eu/
    region=asia/

Partitioning should be based on actual access patterns.

Creating hundreds or thousands of tiny partitions can create its own problems.

Small Files Are a Problem

A common data-lake problem is producing too many small Parquet files.

For example:

orders/
    part-00001.parquet
    part-00002.parquet
    part-00003.parquet
    ...
    part-90000.parquet

Even if the total dataset is not extremely large, managing a huge number of files can increase metadata and query overhead.

File sizes should therefore be considered when designing the ingestion pipeline.

The optimal file size depends on the workload and processing engine, so teams should benchmark with realistic data.

External Data vs ETL

The architectural difference can be summarized as follows.

Traditional ETL

S3 Parquet
     |
     v
Extract
     |
     v
Transform
     |
     v
Load
     |
     v
Aurora Table

External Query Architecture

S3 Parquet
     |
     v
Aurora Query

The second approach can eliminate a data-copy stage.

However, it does not eliminate the need for data preparation. Data still needs to be produced, validated, organized, governed, and maintained.

When ETL Is Still Better

Direct access should not be treated as a universal replacement for ETL.

ETL or ELT can be preferable when:

Data requires significant transformation
Queries are extremely frequent
Very low latency is required
The destination schema is optimized for application access
Data must be denormalized
Complex business logic needs to run before consumption
The workload requires transactional consistency

For example, if an application repeatedly needs the same aggregated metric, calculating it every time from a large Parquet dataset may be inefficient.

A precomputed table or materialized dataset could be a better solution.

Combining Aurora Data With Parquet Data

One interesting use case is combining operational and historical information.

For example:

Aurora
 |
 +---- Current Customer Data
 |
 +---- Current Orders
 |
 +---- Current Account State

S3
 |
 +---- Historical Orders
 |
 +---- Historical Events
 |
 +---- Archived Activity

An analytical workflow may need information from both locations.

Conceptually:

Current Aurora Data
        +
Historical Parquet Data
        |
        v
Combined Query / Analysis

This can be useful when the application needs current operational context together with historical information.

Security Considerations

Querying external data introduces another security boundary.

The database must have appropriate access to the S3 data.

A production design should consider:

IAM permissions
S3 bucket policies
Encryption
Network configuration
Database roles
Data classification
Audit logging
Access boundaries

Use least privilege.

If an application only needs access to:

s3://company-data/orders/

there is little reason to grant access to an unrelated collection of sensitive datasets.

Data Governance

S3 data often survives longer than the application that originally created it.

That makes governance important.

Each dataset should have clear information about:

Dataset
 |
 +---- Owner
 +---- Purpose
 +---- Classification
 +---- Retention
 +---- Schema
 +---- Consumers

For sensitive information, teams should also understand whether direct database access could expose data that the application normally could not access.

Performance Considerations

The biggest performance mistake is assuming that querying Parquet will behave like querying an indexed PostgreSQL table.

The two storage models are fundamentally different.

For large datasets, investigate:

Amount of data scanned
Number of files
File sizes
Partition layout
Columns selected
Filter selectivity
Query frequency
Concurrent workloads

For example, this query may be unnecessarily expensive:

SELECT *
FROM historical_orders
WHERE customer_name LIKE '%smith%';

A more focused query might use a stable identifier and a time restriction:

SELECT
    order_id,
    order_date,
    order_amount
FROM historical_orders
WHERE customer_id = 'CUS-2031'
  AND order_date >= DATE '2026-01-01';

The best query depends on how the data is physically organized.

Protecting Aurora Workloads

Aurora may already be handling application transactions.

Heavy analytical queries should not be allowed to unexpectedly interfere with critical operations.

A production architecture should consider:

Transactional Workload
        |
        v
Aurora

Analytical Workload
        |
        v
Lake / Analytical Engine

If Aurora-based external queries are used, monitor their impact on the database and application.

Do not assume that moving the data outside Aurora automatically makes every query inexpensive.

Common Mistakes

Treating Parquet Like a PostgreSQL Table

Parquet is optimized for analytical data access, not traditional transactional behavior.

Selecting Every Column

Avoid SELECT * when only a few fields are required.

Ignoring File Layout

Poorly organized files can make large datasets expensive to query.

Creating Too Many Small Files

Small-file proliferation can create metadata and processing overhead.

Copying the Entire Dataset Into Aurora

If the application does not require a local copy, copying everything may introduce unnecessary storage and synchronization work.

Running Frequent Heavy Queries

A query that is acceptable once per day may be inappropriate when executed thousands of times per hour.

Troubleshooting

Data Cannot Be Read

Check:

S3 permissions
Database configuration
File location
Parquet schema
Encryption settings
Required database capabilities
Region configuration

Query Is Slow

Start by checking:

Number of files
Data volume
Partitioning
Columns selected
Filter conditions
Query plan
Concurrent database workloads

Schema Errors Occur

Compare the expected schema with the actual Parquet file schema.

Pay attention to:

Data types
Nullable columns
Column names
Date and timestamp types
Schema changes over time

Results Are Incomplete

Check whether:

Files were successfully written
Partition paths are correct
The query includes all relevant partitions
Recent files are visible
Data ingestion has completed

Best Practices

Keep large analytical datasets in S3 when appropriate.
Use Parquet for column-oriented analytical storage.
Select only required columns.
Filter data early.
Design partitions around real query patterns.
Avoid excessive small files.
Monitor query performance with realistic data volumes.
Protect Aurora transactional workloads from expensive analytical queries.
Use least-privilege S3 permissions.
Document Parquet schemas and ownership.
Plan for schema evolution.
Use Iceberg when table-level lake management is required.
Do not assume external queries eliminate data-engineering responsibilities.
Benchmark before moving production workloads away from an existing ETL architecture.

Advantages and Disadvantages

Advantages

Can reduce unnecessary ETL and data duplication
Keeps analytical data in S3
Uses an open and widely supported file format
Works well for large analytical datasets
Can provide access to historical data without loading everything into Aurora
Separates transactional storage from analytical storage

Disadvantages

Query performance depends heavily on data organization
Parquet does not provide full table-management capabilities by itself
Large scans can be expensive or slow
Security spans Aurora and S3
Schema management remains important
Not every workload should bypass ETL

When Should You Query Parquet Without ETL?

This architecture is worth considering when:

Historical data already exists in S3
Data is naturally analytical
Applications need occasional or focused access
Copying the data into Aurora would create unnecessary duplication
The workload can tolerate the characteristics of external data access
The organization wants to reduce pipeline complexity

It is less suitable for high-frequency transactional operations that require predictable low-latency database access.

Summary

Aurora PostgreSQL and Parquet can work together to reduce unnecessary data movement between operational applications and data lakes.

Instead of automatically copying historical data from S3 into Aurora through an ETL pipeline, teams can keep analytical data in its existing Parquet representation and query it through the supported Aurora integration.

The architecture works best when responsibilities remain clear. Aurora handles workloads that benefit from a relational transactional database, while S3 and Parquet handle large analytical datasets.

Performance, security, partitioning, file organization, schema evolution, and workload isolation still need careful attention. Querying Parquet without ETL is not a replacement for every data pipeline, but for the right workload it can remove an entire layer of unnecessary data movement.