Introduction

Parquet has become a common storage format for analytical data because it stores data in a column-oriented structure and works well with large datasets in object storage.

Many applications, however, still use PostgreSQL for operational workloads. This creates a common architecture problem: application data lives in PostgreSQL while historical or analytical data is stored as Parquet files in Amazon S3.

The traditional solution is to build an ETL or ELT pipeline that copies and transforms the Parquet data before it can be queried from a database-oriented application.

Aurora PostgreSQL provides capabilities that can make this architecture simpler by allowing PostgreSQL workloads to work with data stored externally in S3, including Parquet-based datasets.

The result can be a design where the data remains in its original analytical storage location rather than being copied into Aurora simply to make it queryable.

What Is Parquet?

Apache Parquet is a columnar file format designed for efficient analytical workloads.

A traditional row-oriented format stores records approximately like this:

Code
Row 1: customer_id, order_date, amount, status
Row 2: customer_id, order_date, amount, status
Row 3: customer_id, order_date, amount, status

Parquet organizes data by columns:

Code
customer_id
customer_id
customer_id

order_date
order_date
order_date

amount
amount
amount

status
status
status

This layout can be efficient when a query needs only a few columns from a large dataset.

For example:

SQL
SELECT customer_id, amount
FROM historical_orders
WHERE order_date >= DATE '2026-01-01';

A column-oriented format can avoid processing unrelated columns when the underlying query engine supports the appropriate column and predicate pruning.

Why Store Parquet in Amazon S3?

Amazon S3 is commonly used as the storage layer for data lakes.

A simplified structure might look like:

Code
S3
 |
 +---- orders/
 |      |
 |      +---- 2024/
 |      +---- 2025/
 |      +---- 2026/
 |
 +---- customers/
 |
 +---- events/

The actual files may be Parquet:

Code
orders-0001.parquet
orders-0002.parquet
orders-0003.parquet

This approach separates storage from database compute.

Large historical datasets can remain in S3 while transactional data continues to live in Aurora PostgreSQL.

Why Query Parquet From Aurora PostgreSQL?

Consider an application with this architecture:

Code
Aurora PostgreSQL
     |
     +---- customers
     +---- active_orders
     +---- payments

S3
     |
     +---- historical_orders.parquet
     +---- application_events.parquet

Suppose a reporting feature needs five years of historical order information.

One approach is:

Code
Parquet
   |
   v
ETL
   |
   v
Aurora Table
   |
   v
Application Query

That introduces another copy of the data.

With an external-data architecture, the application can access the data where it already exists:

Code
Application
    |
    v
Aurora PostgreSQL
    |
    v
Parquet in S3

This can reduce unnecessary data movement for suitable workloads.

The Architecture

A simplified design looks like this:

Code
                  Application
                       |
                       v
              Aurora PostgreSQL
                       |
                  SQL Query
                       |
                       v
                 Parquet Data
                       |
                       v
                  Amazon S3

The database and object storage serve different purposes.

Aurora can remain responsible for transactional application data, while S3 stores larger analytical datasets.

A Practical Example

Imagine an e-commerce application.

Aurora contains current operational information:

Code
customers
orders
products
payments

S3 contains historical transactions:

Code
historical_orders/
    2024/
    2025/
    2026/

The application wants to calculate historical sales for a customer.

Instead of importing all historical records into Aurora, the relevant Parquet dataset can remain in S3.

A conceptual query could look like:

SQL
SELECT
    customer_id,
    SUM(order_amount) AS total_spend
FROM historical_orders
WHERE order_date >= DATE '2025-01-01'
GROUP BY customer_id;

The exact syntax depends on the Aurora PostgreSQL capability and integration being used.

The architectural idea is that the Parquet data does not have to become a traditional Aurora table simply because an application needs to query it.

Why Parquet Works Well for Analytical Data

Parquet provides several characteristics that make it attractive for lake storage.

Columnar Storage

Analytical queries frequently access a subset of columns.

For example:

SQL
SELECT customer_id, order_amount
FROM historical_orders;

A columnar format is well suited to this type of access.

Compression

Parquet supports compression, reducing storage requirements and potentially reducing the amount of data that needs to be processed.

Schema Information

Parquet files contain schema information describing their columns and data types.

Ecosystem Compatibility

Parquet is supported by many data-processing and analytics systems, making it useful when multiple platforms need to access the same data.

Parquet vs Iceberg

Parquet and Iceberg solve different problems.

Parquet is primarily a file format.

Iceberg is a table format that manages collections of data files and associated metadata.

Feature

Parquet

Iceberg

Type

File format

Table format

Columnar storage

Yes

Usually uses formats such as Parquet

Schema

File-level

Table-level

Table snapshots

No

Yes

Schema evolution

Basic file schema

Advanced table-level management

Partition management

File/layout dependent

Table-level

Historical snapshots

No

Yes

Best use

Individual analytical files

Managed analytical tables

This distinction matters when designing a data lake.

A collection of Parquet files may be enough for simple workloads.

Iceberg becomes more useful when the dataset needs table-level management, snapshots, schema evolution, and other data-lake capabilities.

Querying Only the Data You Need

Large Parquet datasets should not be treated like small database tables.

A query such as:

SQL
SELECT *
FROM historical_orders;

can require processing a significant amount of data.

Instead, select only the necessary columns:

SQL
SELECT
    customer_id,
    order_date,
    order_amount
FROM historical_orders
WHERE order_date >= DATE '2026-01-01';

Then apply selective filters where possible.

This is important for both performance and cost.

Partitioning Parquet Data

Large datasets should generally be organized around common query patterns.

For example:

Code
historical_orders/
    year=2024/
    year=2025/
    year=2026/

A query filtering on the date may then avoid processing irrelevant partitions when the underlying query architecture supports partition pruning.

Another example could be:

Code
events/
    region=us/
    region=eu/
    region=asia/

Partitioning should be based on actual access patterns.

Creating hundreds or thousands of tiny partitions can create its own problems.

Small Files Are a Problem

A common data-lake problem is producing too many small Parquet files.

For example:

Code
orders/
    part-00001.parquet
    part-00002.parquet
    part-00003.parquet
    ...
    part-90000.parquet

Even if the total dataset is not extremely large, managing a huge number of files can increase metadata and query overhead.

File sizes should therefore be considered when designing the ingestion pipeline.

The optimal file size depends on the workload and processing engine, so teams should benchmark with realistic data.

External Data vs ETL

The architectural difference can be summarized as follows.

Traditional ETL

Code
S3 Parquet
     |
     v
Extract
     |
     v
Transform
     |
     v
Load
     |
     v
Aurora Table

External Query Architecture

Code
S3 Parquet
     |
     v
Aurora Query

The second approach can eliminate a data-copy stage.

However, it does not eliminate the need for data preparation. Data still needs to be produced, validated, organized, governed, and maintained.

When ETL Is Still Better

Direct access should not be treated as a universal replacement for ETL.

ETL or ELT can be preferable when:

  • Data requires significant transformation

  • Queries are extremely frequent

  • Very low latency is required

  • The destination schema is optimized for application access

  • Data must be denormalized

  • Complex business logic needs to run before consumption

  • The workload requires transactional consistency

For example, if an application repeatedly needs the same aggregated metric, calculating it every time from a large Parquet dataset may be inefficient.

A precomputed table or materialized dataset could be a better solution.

Combining Aurora Data With Parquet Data

One interesting use case is combining operational and historical information.

For example:

Code
Aurora
 |
 +---- Current Customer Data
 |
 +---- Current Orders
 |
 +---- Current Account State

S3
 |
 +---- Historical Orders
 |
 +---- Historical Events
 |
 +---- Archived Activity

An analytical workflow may need information from both locations.

Conceptually:

Code
Current Aurora Data
        +
Historical Parquet Data
        |
        v
Combined Query / Analysis

This can be useful when the application needs current operational context together with historical information.

Security Considerations

Querying external data introduces another security boundary.

The database must have appropriate access to the S3 data.

A production design should consider:

  • IAM permissions

  • S3 bucket policies

  • Encryption

  • Network configuration

  • Database roles

  • Data classification

  • Audit logging

  • Access boundaries

Use least privilege.

If an application only needs access to:

Code
s3://company-data/orders/

there is little reason to grant access to an unrelated collection of sensitive datasets.

Data Governance

S3 data often survives longer than the application that originally created it.

That makes governance important.

Each dataset should have clear information about:

Code
Dataset
 |
 +---- Owner
 +---- Purpose
 +---- Classification
 +---- Retention
 +---- Schema
 +---- Consumers

For sensitive information, teams should also understand whether direct database access could expose data that the application normally could not access.

Performance Considerations

The biggest performance mistake is assuming that querying Parquet will behave like querying an indexed PostgreSQL table.

The two storage models are fundamentally different.

For large datasets, investigate:

  • Amount of data scanned

  • Number of files

  • File sizes

  • Partition layout

  • Columns selected

  • Filter selectivity

  • Query frequency

  • Concurrent workloads

For example, this query may be unnecessarily expensive:

SQL
SELECT *
FROM historical_orders
WHERE customer_name LIKE '%smith%';

A more focused query might use a stable identifier and a time restriction:

SQL
SELECT
    order_id,
    order_date,
    order_amount
FROM historical_orders
WHERE customer_id = 'CUS-2031'
  AND order_date >= DATE '2026-01-01';

The best query depends on how the data is physically organized.

Protecting Aurora Workloads

Aurora may already be handling application transactions.

Heavy analytical queries should not be allowed to unexpectedly interfere with critical operations.

A production architecture should consider:

Code
Transactional Workload
        |
        v
Aurora

Analytical Workload
        |
        v
Lake / Analytical Engine

If Aurora-based external queries are used, monitor their impact on the database and application.

Do not assume that moving the data outside Aurora automatically makes every query inexpensive.

Common Mistakes

Treating Parquet Like a PostgreSQL Table

Parquet is optimized for analytical data access, not traditional transactional behavior.

Selecting Every Column

Avoid SELECT * when only a few fields are required.

Ignoring File Layout

Poorly organized files can make large datasets expensive to query.

Creating Too Many Small Files

Small-file proliferation can create metadata and processing overhead.

Copying the Entire Dataset Into Aurora

If the application does not require a local copy, copying everything may introduce unnecessary storage and synchronization work.

Running Frequent Heavy Queries

A query that is acceptable once per day may be inappropriate when executed thousands of times per hour.

Troubleshooting

Data Cannot Be Read

Check:

  1. S3 permissions

  2. Database configuration

  3. File location

  4. Parquet schema

  5. Encryption settings

  6. Required database capabilities

  7. Region configuration

Query Is Slow

Start by checking:

  • Number of files

  • Data volume

  • Partitioning

  • Columns selected

  • Filter conditions

  • Query plan

  • Concurrent database workloads

Schema Errors Occur

Compare the expected schema with the actual Parquet file schema.

Pay attention to:

  • Data types

  • Nullable columns

  • Column names

  • Date and timestamp types

  • Schema changes over time

Results Are Incomplete

Check whether:

  • Files were successfully written

  • Partition paths are correct

  • The query includes all relevant partitions

  • Recent files are visible

  • Data ingestion has completed

Best Practices

  1. Keep large analytical datasets in S3 when appropriate.

  2. Use Parquet for column-oriented analytical storage.

  3. Select only required columns.

  4. Filter data early.

  5. Design partitions around real query patterns.

  6. Avoid excessive small files.

  7. Monitor query performance with realistic data volumes.

  8. Protect Aurora transactional workloads from expensive analytical queries.

  9. Use least-privilege S3 permissions.

  10. Document Parquet schemas and ownership.

  11. Plan for schema evolution.

  12. Use Iceberg when table-level lake management is required.

  13. Do not assume external queries eliminate data-engineering responsibilities.

  14. Benchmark before moving production workloads away from an existing ETL architecture.

Advantages and Disadvantages

Advantages

  • Can reduce unnecessary ETL and data duplication

  • Keeps analytical data in S3

  • Uses an open and widely supported file format

  • Works well for large analytical datasets

  • Can provide access to historical data without loading everything into Aurora

  • Separates transactional storage from analytical storage

Disadvantages

  • Query performance depends heavily on data organization

  • Parquet does not provide full table-management capabilities by itself

  • Large scans can be expensive or slow

  • Security spans Aurora and S3

  • Schema management remains important

  • Not every workload should bypass ETL

When Should You Query Parquet Without ETL?

This architecture is worth considering when:

  • Historical data already exists in S3

  • Data is naturally analytical

  • Applications need occasional or focused access

  • Copying the data into Aurora would create unnecessary duplication

  • The workload can tolerate the characteristics of external data access

  • The organization wants to reduce pipeline complexity

It is less suitable for high-frequency transactional operations that require predictable low-latency database access.

Summary

Aurora PostgreSQL and Parquet can work together to reduce unnecessary data movement between operational applications and data lakes.

Instead of automatically copying historical data from S3 into Aurora through an ETL pipeline, teams can keep analytical data in its existing Parquet representation and query it through the supported Aurora integration.

The architecture works best when responsibilities remain clear. Aurora handles workloads that benefit from a relational transactional database, while S3 and Parquet handle large analytical datasets.

Performance, security, partitioning, file organization, schema evolution, and workload isolation still need careful attention. Querying Parquet without ETL is not a replacement for every data pipeline, but for the right workload it can remove an entire layer of unnecessary data movement.