Organizations often store data across multiple cloud platforms. For example, a company might keep its business data in Amazon S3 while using BigQuery for analytics and reporting.
When developers need to analyze this data, they usually consider two options: move the data into Google Cloud or query it directly from its existing storage location.
Moving data can provide good performance, but it also requires storage, data pipelines, synchronization, and maintenance. BigQuery Lakehouse cross-cloud data access offers another approach. It allows BigQuery to query supported Apache Iceberg tables stored in other cloud environments without copying the original data files.
With cross-cloud caching enabled, Lakehouse can also keep frequently accessed data blocks in Google Cloud. Later queries can reuse those blocks instead of repeatedly retrieving them from remote storage.
But does this mean moving data is no longer necessary?
No. Both approaches solve different problems. The right choice depends on how frequently the data is queried, how much data is processed, how quickly results are needed, and how the organization manages its cloud infrastructure.
Let's compare both approaches and understand when Lakehouse caching makes more sense.
Understand the Two Approaches
Before comparing them, let's understand how each approach works.
Moving data into Google Cloud
In this approach, data stored in Amazon S3 or Azure Data Lake Storage is transferred to Google Cloud Storage or loaded into BigQuery.
A typical process looks like this:
- Read the source data from remote storage.
- Transfer the files to Google Cloud.
- Load or process the data.
- Run BigQuery queries against the local data.
- Schedule additional transfers to keep the destination updated.
This approach gives analytics teams control over their data pipelines and destination storage.
It is useful when the organization needs to transform, clean, combine, or maintain a dedicated analytical copy of the source data.
However, the team must manage data freshness, storage, transfer costs, and pipeline failures.
Querying remote data with Lakehouse caching
With Lakehouse cross-cloud data access, BigQuery can query supported remote Apache Iceberg tables without moving the original files.
The process looks different:
- Lakehouse connects to the remote Iceberg catalog.
- It discovers the table metadata.
- BigQuery executes a query against the remote table.
- Required data blocks are retrieved from remote storage.
- When caching is enabled, retrieved blocks are temporarily stored in Google Cloud.
- Subsequent queries can reuse those cached blocks.
The original files remain in the remote cloud.
Google Cloud manages cache capacity, retention, and eviction automatically. Teams do not need to create and maintain a separate cache storage system.
Compare Lakehouse Caching With Data Migration
Both approaches allow BigQuery users to work with data that originally exists outside Google Cloud, but their operational requirements are different.
| Comparison | Lakehouse with cross-cloud caching | Data migration |
|---|
| Original data location | Remains in remote storage | Data is copied to Google Cloud |
| Initial setup | Catalog and cross-cloud connection | Transfer and ingestion pipelines |
| Data synchronization | Reads remote table metadata and data | Requires synchronization processes |
| Storage duplication | No permanent full copy required | Usually requires additional storage |
| Repeated queries | Can reuse cached data blocks | Reads from the destination copy |
| Data transformation | Depends on query and supported table capabilities | Can transform data during ingestion |
| Data freshness | Depends on source updates and metadata synchronization | Depends on pipeline frequency |
| Operational work | Remote access and catalog management | Pipeline and destination management |
There is no universal winner. The decision depends on the workload.
When Lakehouse Caching Is a Better Choice
Lakehouse caching is particularly useful when an organization wants to query remote data regularly but does not want to maintain another permanent copy.
Consider the following situations.
Your data is already stored in Amazon S3 or Azure
Suppose an organization stores several years of transaction history in Amazon S3 using Apache Iceberg.
The analytics team uses BigQuery to generate sales reports.
Moving the entire historical dataset into Google Cloud might create unnecessary storage and synchronization work.
Instead, the team can configure a supported remote catalog and query the Iceberg tables directly.
The data remains in Amazon S3, while BigQuery provides the analytics environment.
This approach is useful when the source data should remain under the control of the existing storage platform.
Multiple queries access the same data
Imagine that a company runs several dashboards against the same sales dataset.
One dashboard shows daily revenue. Another displays product performance. A third analyzes regional sales.
Although the SQL queries are different, they might read many of the same underlying data blocks.
With Lakehouse caching enabled, the first query can retrieve those blocks from remote storage. Later queries can reuse the cached blocks.
This can reduce repeated cross-cloud data transfers.
It is important to remember that caching operates at the data-block level, not by simply storing the final result of an SQL query.
You want to avoid maintaining ETL pipelines
Some organizations maintain multiple pipelines only to make remote data available for reporting.
These pipelines require scheduling, monitoring, retries, and failure handling.
If the organization only needs to query supported Iceberg tables, direct cross-cloud access can remove the need for a separate data-copying process.
This reduces the number of components that the engineering team needs to maintain.
You need to access data from multiple cloud platforms
Large organizations often use different cloud providers for different business requirements.
For example:
- Amazon S3 stores historical transaction data.
- Azure Data Lake Storage holds operational datasets.
- BigQuery handles analytics and reporting.
Lakehouse supports remote catalog integrations such as AWS Glue Data Catalog, Databricks Unity Catalog, and Snowflake Horizon Catalog.
This allows teams to access supported remote tables through a common analytics environment.
When Moving Data Is Still the Better Choice
Lakehouse caching is useful, but it does not replace every data engineering requirement.
There are situations where moving data is the more practical approach.
You run frequent full-table scans
Consider a reporting system that processes several terabytes of data every hour.
If every query reads a different set of files, caching might provide limited benefits.
The system may repeatedly retrieve large amounts of data from remote storage.
In this situation, keeping a local analytical copy in Google Cloud can provide more predictable query performance.
The decision should be based on actual query patterns and transfer costs.
You need extensive data transformation
Suppose your organization receives raw JSON, CSV, and application log files from several systems.
Before reporting, the data needs to be:
- Cleaned and validated.
- Joined with customer information.
- Standardized.
- Deduplicated.
- Aggregated into reporting tables.
Lakehouse cross-cloud access is not a replacement for a complete data transformation pipeline.
You might still need to ingest and process the source data using Dataflow, Dataform, Apache Spark, or another data engineering platform.
Your application requires predictable low latency
Remote queries depend on network connectivity, data location, and the amount of data retrieved.
A private Cross-Cloud Interconnect can provide more predictable network performance, but it does not make remote storage identical to local storage.
If an application has strict latency requirements and executes large numbers of queries, maintaining a local copy might be a better option.
Your organization has strict data residency requirements
When Lakehouse caching is enabled, retrieved data blocks are temporarily stored in the Google Cloud region where the query executes.
For example, an organization might store customer data in an AWS region in Europe but run BigQuery queries in a US region.
Before enabling caching, the organization must verify that temporary cached copies in the query region comply with its data residency requirements.
If those requirements do not allow this arrangement, moving data to an approved destination or querying without caching might be more appropriate.
Understand the Cost Difference
Cost is one of the main reasons organizations consider Lakehouse caching.
However, comparing only storage costs is not enough.
With direct cross-cloud access, you should consider:
- Remote cloud data egress charges.
- Cross-Cloud Interconnect costs, if used.
- BigQuery query processing costs.
- Network infrastructure.
- Remote catalog operations.
- Data access frequency.
With data migration, consider:
- Source cloud egress.
- Destination storage.
- Data transfer services.
- Pipeline execution.
- Data synchronization.
- BigQuery query processing.
Google documents cross-cloud caching as having no additional charge. However, the initial remote reads and cache misses can still incur data transfer charges from the source cloud.
Example: Repeated access to the same dataset
Suppose a company runs several reports against the same remote dataset.
For illustration, assume the workload reads 100 GB of unique file blocks during the first query. Subsequent reports need the same blocks.
Without caching, the remote storage might need to transfer those blocks repeatedly.
With caching, later queries can read the blocks locally, provided they remain in the cache and the queries access the same data.
The actual savings depend on:
- How often queries access the same blocks.
- How frequently source files change.
- How long the blocks remain cached.
- How much data each query reads.
- The source cloud's egress pricing.
Do not assume that every repeated query will produce a cache hit. Measure actual usage before estimating savings.
Measure Whether Caching Is Working
BigQuery exposes cross-cloud cache statistics through the objectStorageStats field in query job statistics.
Two useful metrics are:
| Metric | Description |
|---|
cacheBytesRead | Data read from the local Google Cloud cache |
objectStorageBytesRead | Data read directly from remote object storage |
You can inspect these values in the BigQuery console or retrieve them through the BigQuery API.
For example, suppose a query reports:
cacheBytesRead: 80 GB
objectStorageBytesRead: 20 GB
This means 80 GB was read from the local cache and 20 GB was read directly from remote storage.
The total is 100 GB across these two sources.
For a simple cache-read ratio:
[
\text{Cache Read Ratio} =
\frac{\text{Cache Bytes Read}}
{\text{Cache Bytes Read}+\text{Remote Bytes Read}}
\times 100
]
Using the example:
[
\frac{80}{80+20}\times100=80%
]
An 80% cache-read ratio means that 80% of the bytes reported across these two sources came from the local cache.
Track this metric across several query runs. A single query is not enough to understand the behavior of a production workload.
A Practical Decision Guide
Before choosing an approach, answer a few questions about your application.
| Requirement | Better starting point |
|---|
| Query remote Iceberg data without copying it | Lakehouse |
| Repeatedly access the same remote data blocks | Lakehouse with caching |
| Avoid maintaining a separate ingestion pipeline | Lakehouse |
| Transform and clean large volumes of raw data | Data migration and processing |
| Run predictable, high-volume analytical workloads | Evaluate a local analytical copy |
| Keep cached data within specific geographic boundaries | Evaluate regional requirements first |
| Maintain a dedicated reporting dataset | Data migration |
You can also combine both approaches.
For example, an organization can query historical Iceberg data directly through Lakehouse while maintaining a local BigQuery table for frequently updated operational reports.
This avoids treating every dataset in the same way.
Summary
BigQuery Lakehouse caching is a useful option when organizations want to query supported remote Apache Iceberg tables without maintaining permanent copies in Google Cloud. It works particularly well when multiple queries access the same data blocks and teams want to reduce repeated cross-cloud transfers. However, data migration remains useful for large transformation pipelines, high-volume workloads, and applications that require predictable local query performance. The best decision comes from measuring query patterns, data transfer, storage costs, latency, and compliance requirements. In many environments, combining remote access with selective data migration provides the most practical solution.
Join the conversation! Your thoughts help the community grow.