BigQuery Adds Incremental Data Transfers for PostgreSQL and ServiceNow

Many companies keep their operational data in different systems. Customer orders might live in PostgreSQL, while support tickets, incidents, and change requests are managed in ServiceNow. When a business wants to analyze this information in BigQuery, it needs a reliable way to bring the data together.

The challenge is that business data changes constantly. A support ticket may receive a new status, an order may be updated, or a new customer record may be created. If a data pipeline copies every record during every scheduled run, it can waste time and resources processing information that has not changed.

Google has added incremental data transfer support for PostgreSQL and ServiceNow through BigQuery Data Transfer Service. Announced as a Preview capability in April 2026, this feature allows eligible transfer configurations to retrieve new and updated records instead of repeatedly loading the entire source dataset.

For developers and data engineers, this can make recurring data pipelines more efficient. However, incremental transfers require careful configuration. The source table needs a suitable watermark column, and the destination must use an appropriate write mode to avoid duplicate or outdated records.

What Is Incremental Data Transfer?

A full data transfer copies the complete selected dataset from the source system during each run. An incremental transfer focuses on records that have changed since the previous successful synchronization point.

Consider a PostgreSQL table containing five million orders. A scheduled transfer runs every hour, but only 2,000 orders have changed since the previous run.

With a full transfer, the pipeline repeatedly processes the entire dataset. With a correctly configured incremental transfer, it can focus on the changed records.

The same idea applies to ServiceNow. An organization might have hundreds of thousands of incident records, but only a small portion receives updates during a typical hour.

Incremental transfers reduce unnecessary data movement and can make recurring ingestion jobs more practical. They do not eliminate the initial data load, though. When incremental ingestion is enabled for the first time, the first transfer loads all available data. Subsequent runs transfer new and updated records.

Which Data Sources Are Supported?

The new capability applies to two useful data sources.

Data source	Typical data	Main benefit
PostgreSQL	Orders, customer records, transactions, application data	Keep analytical tables updated without repeatedly reloading the entire dataset
ServiceNow	Incidents, change requests, service records, IT operations data	Analyze recent operational changes without repeatedly transferring all records

PostgreSQL can run on premises, in Cloud SQL, or with other cloud providers. This makes the feature relevant to teams that want to analyze data in BigQuery without first migrating their operational database.

ServiceNow is particularly useful for organizations that want to analyze IT service management data. Teams can use BigQuery to examine incident trends, resolution times, recurring problems, and changes in service workloads.

Incremental ingestion is currently a Preview feature, so teams should evaluate its limitations and support requirements before relying on it for critical production reporting.

How Incremental Transfers Work

The basic process is straightforward, but two settings determine whether the resulting data stays accurate: the watermark column and the write mode.

1. BigQuery identifies changed records

A watermark column records when a row was created or modified. During an incremental transfer, the connector uses this column to identify the records that need to be transferred.

For example, a PostgreSQL table might contain the following fields:

CREATE TABLE orders (
    order_id BIGINT PRIMARY KEY,
    customer_id BIGINT NOT NULL,
    order_status VARCHAR(30) NOT NULL,
    total_amount NUMERIC(12, 2) NOT NULL,
    created_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP NOT NULL DEFAULT CURRENT_TIMESTAMP
);

Here, updated_at is a possible watermark column because it is intended to change whenever an order changes.

However, declaring a column does not automatically keep it updated. The application, a database trigger, or another database mechanism must maintain the value whenever the row is modified.

For an incremental transfer that must capture updates, a column that changes only during insertion is not sufficient.

2. The connector transfers the relevant rows

The connector uses the configured watermark to identify eligible records. The source data is then transferred to the destination BigQuery table.

The watermark must meet the connector's requirements. PostgreSQL incremental transfers support TIMESTAMP watermark columns, while ServiceNow incremental transfers require DATETIME watermark columns. The values must also be monotonically increasing, so the selected field must behave correctly for the connector's change-tracking process.

A timestamp column that is frequently reset, populated incorrectly, or left unchanged after updates can cause problems. Developers should verify the behavior of the column before enabling incremental ingestion.

3. BigQuery writes the transferred data

Finding changed records is only part of the job. BigQuery must also know what to do with them.

The connector supports different write strategies depending on the source and configuration. PostgreSQL supports Append and Upsert modes. ServiceNow incremental ingestion is configured with Upsert in the current setup instructions.

These choices affect whether the destination contains repeated rows or the latest version of each record.

Append vs. Upsert: Which Should You Choose?

The write mode determines how incoming records are handled in BigQuery.

Append mode

Append mode inserts the transferred rows without checking whether the same records already exist in the destination table.

This approach is useful when each incoming row represents a new event that should be retained. It can also work for source tables that are effectively insert-only.

For example, an application might collect immutable event records. Each event has its own unique identifier and is never modified after creation.

The problem appears when a source record changes. If an existing order is transferred again, Append mode can add another row for the same order instead of updating the earlier row.

That can lead to duplicate records and incorrect reports unless the downstream query deliberately handles them.

Upsert mode

Upsert mode checks the configured primary key to decide whether to insert a new record or update an existing one.

Suppose an order with ID 10025 changes from Pending to Shipped. If the destination already contains that order and the primary key is configured correctly, Upsert updates the destination row rather than adding a second copy.

For Upsert, you must configure both a watermark column and one or more primary-key columns. The selected keys must be unique and non-null across the source records.

The following example illustrates the intended result:

Order ID	Previous status	New status	Result in BigQuery
10025	Pending	Shipped	Existing row updated
10026	No existing row	Created	New row inserted
10027	Processing	Processing	No change if the record is not selected for transfer

The final row illustrates an important point: incremental ingestion is not a guarantee that every source row will be checked during every run. The connector transfers records identified by its incremental change-tracking process.

For mutable business tables, Upsert is usually the better fit because it keeps the destination closer to the current source state.

Configuring Incremental Transfers for PostgreSQL

You can configure a PostgreSQL transfer through the Google Cloud console or the BigQuery command-line tool.

Before creating the transfer, check that:

The source database is reachable from the transfer service.
The connection uses an appropriate TLS configuration.
The selected watermark column is a valid TIMESTAMP and behaves correctly when records change.
The primary key uniquely identifies each row if you plan to use Upsert.
The destination BigQuery dataset exists and has the required permissions.

In the Google Cloud console, create a PostgreSQL transfer, choose the source connection, select the tables to transfer, and choose Incremental as the ingestion type. Then select Append or Upsert as the write mode.

For Upsert, select the watermark column and primary key. For Append, select a watermark column that is appropriate for identifying newly created records.

The following command illustrates the parameters used when creating an incremental PostgreSQL transfer with Upsert. Replace the example connection details and identifiers with values from your environment.

bq mk \
  --transfer_config \
  --target_dataset=analytics \
  --data_source=postgresql \
  --display_name="Incremental Orders Transfer" \
  --params='{
    "assets": ["sales.public.orders"],
    "ingestionType": "incremental",
    "writeMode": "WRITE_MODE_UPSERT",
    "watermarkColumns": ["updated_at"],
    "primaryKeys": [["order_id"]],
    "connector.database": "sales",
    "connector.endpoint.host": "postgres.example.internal",
    "connector.endpoint.port": 5432
  }'

This is an illustrative configuration, not a complete connection setup. The PostgreSQL connector also requires the appropriate authentication, network connectivity, and TLS settings. Check the required parameters for your connection type before running the command.

The assets entry identifies the source table. The watermark column identifies changes, and the primary key tells BigQuery which destination row should be updated.

One important consideration is that the initial incremental run still loads all available source data. Plan for that first load when estimating database load, transfer duration, and the time required to populate the destination table.

Configuring Incremental Transfers for ServiceNow

ServiceNow contains records such as incidents, change requests, and other operational data. Many of these records change several times during their lifetime, which makes Upsert a useful choice for maintaining an analytical copy.

In the Google Cloud console, create a ServiceNow transfer and provide the required connection and OAuth credentials. Select the tables to transfer, choose Incremental ingestion, and select Upsert as the write mode.

For each selected table, choose a suitable watermark column and primary key. ServiceNow commonly provides sys_updated_on for tracking record updates and sys_id for uniquely identifying records.

A simplified command showing the incremental settings looks like this:

bq mk \
  --transfer_config \
  --target_dataset=it_analytics \
  --data_source=servicenow \
  --display_name="Incremental ServiceNow Incidents" \
  --params='{
    "assets": ["incident"],
    "ingestionType": "incremental",
    "writeMode": "WRITE_MODE_UPSERT",
    "watermarkColumns": ["sys_updated_on"],
    "primaryKeys": [["sys_id"]],
    "connector.instanceId": "YOUR_INSTANCE_ID"
  }'

This command demonstrates the ingestion settings, not a complete working connection. You must also provide the required username, password, OAuth client ID, OAuth client secret, and any required network attachment.

If you transfer multiple ServiceNow tables, the watermark and primary-key entries must correspond to the tables in the assets list in the same order.

For example, if the transfer includes both incident and change_request, configure the corresponding watermark columns and primary keys for both tables. Do not assume that every ServiceNow table has identical fields or update behavior.

Important Limitations Developers Should Know

Incremental ingestion can save work, but it is not a universal replacement for full data transfers.

Deletes are not synchronized

Incremental transfers do not synchronize source-table delete operations. If a record is deleted in PostgreSQL or ServiceNow, the corresponding row can remain in BigQuery.

This matters when reports must reflect the exact current state of the source. You may need a separate deletion-tracking process, a soft-delete flag, or periodic reconciliation to keep the destination accurate.

Watermark columns must be reliable

The watermark column is central to change tracking. If a row changes but the selected timestamp does not, the connector might not recognize that change.

For PostgreSQL, a column such as updated_at must be maintained correctly. For ServiceNow, choose a suitable DATETIME field that reflects record modifications and satisfies the connector's requirements.

Configuration changes can be restricted

After the first incremental run, you cannot freely change the selected source objects, write mode, watermark column, or primary key in the same transfer configuration. Plan these settings before starting the transfer.

If the source schema changes, the destination schema may also change, but not every source change is handled identically. For example, a renamed source column can result in the original destination column remaining while a new column is added.

Incremental ingestion requires careful validation

A successful transfer does not automatically prove that the destination is a perfect copy of the source. Incorrect keys, unreliable watermarks, missing deletes, and source schema changes can all affect the results.

Run reconciliation checks before depending on the destination for financial reporting, operational dashboards, or business-critical decisions.

Advantages and Disadvantages

Advantages

Less repeated data movement: Later runs can focus on new and updated records rather than repeatedly loading the entire dataset.
More practical scheduled pipelines: Frequent transfers can be more efficient when only a small portion of the source data changes.
Managed ingestion: BigQuery Data Transfer Service handles the scheduled transfer process without requiring a separate custom extraction service.
Better destination maintenance with Upsert: Existing records can be updated instead of duplicated when the primary key is configured correctly.
Useful for analytics: Teams can bring application and IT service data together for reporting and analysis in BigQuery.

Disadvantages

Preview status: The capability may change, and teams should review the current support and limitations.
Initial full load: The first incremental transfer still processes all available data.
No automatic delete synchronization: Deleted source records can remain in BigQuery.
Strict configuration requirements: Watermark columns and primary keys must be chosen carefully.
Schema changes require attention: Renamed columns and unsupported type conversions can create inconsistencies or cause a transfer to fail.

Best Practices for Reliable Incremental Transfers

Choose the watermark column based on how the data changes. If existing rows can be updated, use a column that reflects those updates. Do not choose a creation timestamp simply because it is available.

Use stable primary keys. The selected key must uniquely identify each record. For a mutable business table, an incorrect key can update the wrong row or produce inconsistent destination data.

Create an index on the PostgreSQL watermark column where appropriate. Incremental transfers use the watermark in source queries. An index can help the database find qualifying records more efficiently.

Schedule transfers with enough time to finish. Avoid overlapping runs for the same configuration. For ServiceNow, overlapping jobs can be skipped, and multiple concurrent transfers against the same instance can increase load.

Monitor failures and transfer duration. Enable notifications and investigate missed runs. A delayed transfer can affect dashboards even if the previous transfer completed successfully.

Keep a deletion strategy. If reports require the destination to match the source exactly, design a separate method for detecting deleted records.

Test before switching critical workloads. Validate row counts, key uniqueness, updated values, schema changes, and deletion behavior before making the incremental pipeline the only source of analytical data.

Conclusion

Incremental data transfers for PostgreSQL and ServiceNow give developers a more efficient way to keep BigQuery datasets updated. Instead of repeatedly transferring every record, the connectors can focus on new and modified data after the initial load.

The biggest benefit is reduced unnecessary data movement. But reliable results depend on more than enabling an incremental setting. Developers must select the right watermark column, choose the appropriate write mode, use stable primary keys, and understand how deletes and schema changes are handled.

For PostgreSQL, both Append and Upsert are available. For ServiceNow, the current incremental setup uses Upsert. In either case, test the configuration with real data and verify the destination before relying on it for business-critical analytics.

For teams that regularly analyze application data or IT service records, this Preview feature is worth testing. A carefully configured incremental transfer can simplify the ingestion pipeline while keeping analytical data fresher and reducing the work performed during recurring runs.