Build Delta Lake Imports With Dataflow Job Builder

The source configuration also provides an optional field for Hadoop configuration properties. If your environment requires additional properties to access Cloud Storage, configure them there.

For example, the documented configuration can include:

fs.gs.impl=com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem

Only add properties that your environment requires. A Hadoop filesystem setting should not be treated as a replacement for the correct service-account permissions or Cloud Storage access configuration.

After entering the source details, confirm the configuration.

Step 4: Review the destination

Expand the WriteToIceberg sink configuration in the Sinks section.

The import blueprint typically prepopulates information such as the destination Lakehouse table, catalog, and warehouse location.

Verify that the destination points to the intended catalog, namespace, and table before running the job.

For an existing table, confirm that its schema is compatible with the Delta Lake source. For a new table, review the expected schema after the import completes.

A destination mistake can be more costly than a source configuration error because the pipeline might successfully process data into an unintended table.

Step 5: Run the batch pipeline

After reviewing the source and destination, open the Dataflow options section and select Run job.

Dataflow creates the batch job and displays its execution graph. You can monitor the job status and inspect the individual processing stages.

A successful submission does not mean that the import has completed successfully. Wait for the job to reach a terminal status and confirm that it reports Succeeded.

If the job fails, inspect the job logs and worker logs before retrying. Repeatedly launching the same pipeline without understanding the error can waste processing time and complicate troubleshooting.

Validate the Imported Iceberg Table

After the Dataflow job succeeds, verify that the destination table contains the expected data.

The Lakehouse integration documentation describes querying the imported table through BigQuery. The table can be addressed using its project, catalog, namespace, and table identifiers.

For example, the following query illustrates a basic inspection:

SELECT *
FROM `my-project.my-catalog.my-namespace.orders`
LIMIT 10;

Replace the example identifiers with the actual values for your environment. Confirm the identifier format supported by the destination before using the query.

A successful query establishes that the table can be accessed through the query interface, but it does not prove that every record was imported correctly.

For production imports, perform additional checks:

• Compare source and destination row counts where a reliable source count is available.

• Verify key columns and representative records.

• Check null values, numeric precision, timestamps, and nested fields.

• Confirm that expected partitions or business date ranges are present.

• Compare important aggregates, such as order totals or event counts.

• Run representative downstream queries used by reports and applications.

For large datasets, avoid repeatedly scanning the entire table just to validate a migration. Use partition-aware checks and targeted comparisons where possible.

Handle Schema Compatibility Carefully

Schema compatibility is one of the most important considerations in Delta Lake imports.

Suppose the source table contains these fields:

order_id       STRING
customer_id    STRING
order_total    DECIMAL
created_at     TIMESTAMP

The destination table must represent these fields compatibly for the import to succeed.

Potential problems include a destination missing a source field, incompatible data types, unexpected nullability, or differences in how nested data is represented.

Existing destination schemas are not automatically modified by this workflow. If the destination table has an incompatible schema, review the mismatch and decide whether to correct the destination or create a suitable new table.

Avoid changing a production schema solely to make an import pass without understanding the downstream impact. A change in numeric precision, timestamp interpretation, or nullable fields can affect reports and applications even if the pipeline finishes successfully.

For repeatable imports, treat the source and destination schemas as part of the migration contract. Validate them before launching the job and record intentional differences.

Common Problems and Troubleshooting

The source table cannot be read

First, verify that the source URI points to the Delta Lake table root. Confirm that the _delta_log/ directory and the required data files exist.

Next, check that the Dataflow worker identity has the required access to the Cloud Storage objects.

A directory containing only Parquet files is not sufficient evidence that it is a valid Delta Lake table.

The job fails with a permission error

Check the permissions of the identity that runs the Dataflow workers, not only the account used to configure the job.

The pipeline needs access to the source data and the destination resources. The required permissions depend on the project's resource configuration.

Review the error logs to determine whether the failure involves Cloud Storage, Dataflow, the destination catalog, or another dependency.

The destination schema is incompatible

For an existing table, compare its schema with the source table's schema and correct the incompatibility before rerunning the import.

Do not expect the import workflow to modify the existing destination schema automatically.

The job succeeds, but the results appear incorrect

Check whether the configured source path identifies the intended table, whether the destination is the expected table, and whether representative records match.

Also verify the source's committed Delta Lake state and investigate any differences in data types or timestamp interpretation.

A successful job status is necessary, but data-quality validation remains the responsibility of the migration workflow.

The pipeline is expensive or takes longer than expected

Large imports require processing resources and can generate substantial data-read and data-write activity. Review the Dataflow job graph, worker logs, dataset size, and processing behavior.

Avoid unnecessary repeated imports when a validation query or smaller representative dataset would answer the immediate question.

For recurring ingestion or highly customized transformations, evaluate whether a reusable Dataflow pipeline or an Apache Beam implementation is more appropriate than repeatedly configuring a one-off import.

When to Use Job Builder Instead of Custom Apache Beam Code

Job Builder is a strong fit when the import follows a supported source-to-destination pattern and the pipeline does not require extensive custom logic.

It offers a visual workflow, supports pipeline configuration through the console, and can save supported pipeline definitions as Apache Beam YAML for reuse.

A custom Apache Beam pipeline may be more suitable when the migration requires specialized transformations, complex validation rules, custom error handling, or source and destination connectors that the Job Builder does not support.

The decision should be based on the complexity of the data workflow, not simply on whether a no-code option exists.

For a straightforward Delta Lake-to-Lakehouse import, Job Builder reduces implementation effort. For a migration with substantial business logic or strict data-quality requirements, additional pipeline code and automated validation may be necessary.

Summary

Google Cloud Dataflow Job Builder provides a low-code way to import Delta Lake tables stored in Cloud Storage into Lakehouse for Apache Iceberg using a batch pipeline.

The workflow starts by selecting the destination table, configuring the Delta Lake source root, reviewing the Iceberg sink, and running the Dataflow job. New destination tables receive an automatically created schema, while existing destination tables must already have a compatible schema.

The main constraints are that the source must be a valid Delta Lake table in Cloud Storage, the workflow requires a batch job, and Amazon S3 is not supported as the source for this import path.

Before relying on the imported data, verify job completion, inspect the destination table, compare representative records, and validate the schema and downstream queries. These checks turn a convenient visual import into a dependable data migration process.