Apache Iceberg has made it much easier to build analytical tables on object storage, but there is another component that data platforms need to think about: the catalog.
An Iceberg table is not just a collection of Parquet files. Engines need to know where a table lives, which metadata file represents its current state, how the schema has evolved, and how concurrent changes should be handled.
That information is normally provided by an Iceberg catalog.
Google Cloud has introduced a new capability that allows Spanner to act as a runtime catalog for Apache Iceberg tables. The catalog information is stored in Spanner while the actual Iceberg table data and metadata remain in object storage.
The architecture is interesting because it combines two very different systems:
Apache Iceberg
|
+---- Table Data
| |
| v
| Object Storage
|
+---- Catalog
|
v
SpannerSpanner does not become the storage engine for the analytical data.
It becomes the transactional control plane that helps applications and query engines discover and manage Iceberg tables.
Why Iceberg Needs a Catalog
An Iceberg table typically consists of data files plus metadata.
A simplified table might look like:
orders/
data/
file-001.parquet
file-002.parquet
metadata/
v1.metadata.json
v2.metadata.json
v3.metadata.jsonA query engine needs to determine which metadata represents the current table state.
It also needs to understand things such as:
Table location
Schema
Partition specification
Snapshots
Current snapshot
Metadata files
Table propertiesThat information is managed through an Iceberg catalog.
Without a catalog, each engine would need to discover and manage table metadata independently.
The catalog provides a consistent namespace and a mechanism for locating the current table metadata.
What Spanner Adds
Spanner is a globally distributed relational database designed around strong consistency and horizontal scalability.
Using it as an Iceberg catalog brings a transactional database into the control plane of the data lake.
The architecture can be viewed as two separate layers:
Data Plane
------------------
Iceberg Data Files
|
v
Object Storage
Control Plane
------------------
Spanner
|
v
Iceberg CatalogThis separation is useful.
The analytical data can remain in inexpensive object storage while Spanner manages the transactional information required to coordinate access to the tables.
Google describes this architecture as using Spanner's transactional capabilities to provide a reliable catalog for Iceberg tables while keeping the actual table data in object storage.
Why Use a Database as the Catalog?
At first glance, a catalog does not seem like something that needs a globally distributed relational database.
But modern data platforms increasingly have many writers and readers.
Consider:
Iceberg Catalog
|
+------------+------------+
| | |
v v v
Spark Trino Flink
| | |
+------------+------------+
|
v
Object StorageMultiple engines may be reading and writing the same tables.
The catalog therefore becomes an important coordination point.
If two writers try to update the same table simultaneously, the system needs a reliable way to coordinate those changes.
Transactional semantics can make this easier to manage.
That is where Spanner becomes interesting.
Strong Consistency at the Catalog Layer
Spanner provides externally consistent transactions and strong consistency across its distributed architecture.
Using those characteristics for the Iceberg catalog can help coordinate metadata operations.
Imagine two processes attempting to update a table:
Writer A
|
+---- Update Table
|
v
Spanner
Writer B
|
+---- Update Same Table
|
v
SpannerThe catalog can use transactional behavior to coordinate the updates rather than allowing both writers to blindly publish conflicting table states.
This is particularly relevant in data platforms where many pipelines operate concurrently.
The data files remain in object storage, but the catalog provides a strongly managed control point.
Runtime Catalog vs Data Storage
One important distinction is that Spanner does not replace the object store.
An Iceberg table might still use:
Object Storage
|
+---- Parquet data
+---- Iceberg metadata
+---- Manifest files
+---- Delete filesSpanner maintains the catalog information that tells clients how to locate and interact with those tables.
That separation is useful because different storage technologies are optimized for different jobs.
Object storage is well suited to large analytical files.
Spanner is well suited to transactional metadata and coordination.
The architecture therefore becomes:
Query Engine
|
v
Spanner
Catalog
|
v
Iceberg Metadata
|
v
Object Storage
|
v
Data FilesWhy This Matters for Multi-Engine Analytics
One of Iceberg's major advantages is that it is not tied to a single query engine.
Organizations can use different engines for different workloads.
For example:
Iceberg Table
|
+----------+----------+
| | |
v v v
Spark Trino Flink
| | |
+----------+----------+
|
v
Object StoreA common catalog gives those engines a shared view of the table namespace.
This is especially useful when one organization uses Spark for batch processing, Flink for streaming, and another engine for interactive SQL.
The catalog becomes a common control plane rather than forcing every engine to maintain its own independent table registry.
The Catalog Is Part of the Data Architecture
Data engineers sometimes focus heavily on:
Storage
Compute
Networkand treat the catalog as configuration.
That can be a mistake.
The catalog influences:
Table discovery
Concurrency
Schema evolution
Snapshot management
Data access
Engine interoperability
GovernanceIf the catalog becomes unavailable, applications may still have the underlying data files, but discovering and correctly interpreting the current table state becomes much harder.
That makes catalog availability an architectural concern.
A Practical Example
Imagine a company operating a large customer-events platform.
The raw events are stored as Iceberg tables:
customer_events
|
+---- event_time
+---- customer_id
+---- event_type
+---- payloadThe actual Parquet files live in object storage.
Several pipelines interact with the table:
Streaming Pipeline
|
v
Iceberg
Batch ETL
|
v
Iceberg
Analytics
|
v
Iceberg
ML Pipeline
|
v
IcebergSpanner provides the catalog:
Spanner
|
Iceberg Catalog
|
+-----------+-----------+
| | |
Spark Flink SQL
| | |
+-----------+-----------+
|
v
Object StoreNow the table's control metadata is handled through a transactional system while the data remains in object storage.
Concurrency Is a Major Reason to Care
Imagine a table receives updates from several pipelines.
At roughly the same time:
Pipeline A -> Add Data
Pipeline B -> Update Schema
Pipeline C -> Compact TableThese operations can touch related table metadata.
A catalog needs to make sure that readers do not end up seeing an invalid or partially committed table state.
Iceberg itself uses optimistic concurrency control for table updates.
A catalog built on a transactional database can provide a strong foundation for the coordination required around those operations.
The important point is that Spanner does not change Iceberg's fundamental table format.
It provides a transactional implementation for the catalog layer.
Global Applications Are Another Interesting Use Case
Spanner's distributed architecture is particularly relevant when an organization operates across geographic regions.
An application may have workloads such as:
US
EU
Asiawhile still needing a consistent catalog.
The catalog may need to answer:
Where is this table?
Which metadata is current?
What is the current snapshot?
Which schema version should readers use?A globally distributed transactional database can be useful when those operations need strong consistency across locations.
That does not mean every data lake needs a globally distributed catalog.
If the workload is small and regional, the additional infrastructure may provide little benefit.
The value increases when the data platform itself is distributed.
Catalog and Data Residency
There is another architectural consideration.
If the data files are stored in a specific geography, the catalog metadata may still contain information about those datasets.
That means teams should evaluate the catalog's own data location and governance requirements.
A sovereignty-aware architecture therefore needs to consider:
Data files
|
+---- Region
|
+---- Encryption
Catalog
|
+---- Region
|
+---- Access
|
+---- Audit
Processing
|
+---- Region
|
+---- IdentityThe fact that the data itself stays in one location does not automatically mean the entire data platform satisfies the same residency requirements.
The catalog is part of the platform.
Schema Evolution Still Belongs to Iceberg
Spanner does not replace Iceberg's schema management.
Suppose the original table contains:
customer_id
event_type
event_timeLater, the schema evolves:
customer_id
event_type
event_time
device_typeIceberg remains responsible for representing the table schema and its evolution.
Spanner's role is to provide the catalog layer that points clients to the appropriate Iceberg metadata.
That separation is worth keeping clear:
Iceberg
-> Table semantics
-> Schema
-> Snapshots
-> Metadata
Spanner
-> Catalog
-> Namespace
-> Transactional coordination
Object Storage
-> Data files
-> Metadata filesEach layer has a different responsibility.
Why This Could Simplify Data Platforms
One reason organizations adopt Iceberg is to avoid locking analytical data into a proprietary table format.
But open table formats can create their own operational questions.
Teams need to decide:
Which catalog?
How is it hosted?
How is it secured?
How does it scale?
How is concurrency handled?
How do engines connect?A managed Spanner-backed catalog can reduce some of that infrastructure work.
Instead of building a custom catalog service, organizations can use a transactional database designed for distributed workloads.
That does not eliminate operations completely, but it can move the responsibility toward a familiar managed database model.
Common Mistakes
Thinking Spanner Stores the Iceberg Data
It does not.
The actual analytical data remains in object storage.
Spanner provides the catalog layer.
Treating the Catalog as Optional Infrastructure
The catalog is part of how engines discover and coordinate access to Iceberg tables.
Its availability and consistency can directly affect the data platform.
Ignoring Catalog Permissions
A user who can discover a table does not necessarily need unrestricted access to the underlying data.
Catalog permissions and data permissions should be designed separately.
Assuming Every Iceberg Engine Behaves the Same Way
Different engines can have different levels of support for Iceberg features and catalog implementations.
Test the complete combination used by the organization.
Choosing a Distributed Catalog Without a Distributed Requirement
Spanner brings useful distributed and transactional capabilities.
If the data platform is small and regional, those capabilities may not justify the additional architecture.
Advantages and Disadvantages
Advantages
The catalog gets transactional infrastructure. Spanner provides strong consistency and distributed transaction capabilities that are useful for coordinating catalog operations.
Iceberg data remains in object storage. Organizations can continue using object storage for the large analytical data while using Spanner only for the control plane.
Multiple engines can share a catalog. Spark, Flink, Trino, and other Iceberg-compatible engines can use the same table namespace when configured appropriately.
The architecture separates control and data planes. This makes it easier to reason about the responsibilities of storage, metadata, and transactional coordination.
Distributed data platforms get a globally capable catalog option. Spanner's architecture can be useful for organizations operating data workloads across multiple geographic locations.
Disadvantages
It adds another infrastructure component. Teams now need to understand and operate the relationship between Spanner, Iceberg, and object storage.
Spanner may be unnecessary for smaller workloads. A simple regional data platform may not need the distributed transactional capabilities of Spanner.
Compatibility still matters. Every query engine and data-processing system needs to support the chosen Iceberg catalog implementation correctly.
Catalog availability becomes another dependency. If the catalog is unavailable, discovering current table state can become difficult even when the underlying data files remain accessible.
When a Spanner-Based Iceberg Catalog Makes Sense
This architecture is most interesting for organizations building large, distributed analytical platforms with several Iceberg consumers.
Typical examples include:
Large data lakes
Multi-engine analytics
Distributed data platforms
Streaming and batch workloads
Global analytical applications
AI and ML data platforms
Organizations already using SpannerIt is less compelling for a small data lake with one processing engine and a simple catalog requirement.
In that environment, a simpler catalog may be easier to operate and cheaper to maintain.
Summary
Using Spanner as a runtime catalog for Apache Iceberg creates an interesting separation between the data plane and the control plane.
The large analytical files remain in object storage. Iceberg continues to define table semantics, schemas, snapshots, and metadata. Spanner provides the transactional catalog layer that helps applications and query engines discover and coordinate access to those tables.
That separation is the key architectural idea.
The value is not that Spanner becomes a faster place to store Parquet files. It is that a distributed transactional database can provide a strong foundation for the metadata and coordination layer that sits above those files.
For organizations running multiple analytics engines, distributed pipelines, and globally deployed data workloads, the catalog can become an important piece of infrastructure rather than an afterthought.
The trade-off is complexity.
Teams should choose a Spanner-backed Iceberg catalog because the workload actually benefits from distributed consistency, transactional catalog operations, and shared access across engines, not simply because both Spanner and Iceberg are capable technologies.
For large data platforms, though, the architecture points toward an increasingly important pattern: keep analytical data in scalable object storage while using a strongly consistent database to manage the control plane around it.
Join the conversation! Your thoughts help the community grow.