Suppose you have a Lakeflow pipeline that reads sales data from a development catalog and creates a daily sales summary. When you move the same pipeline to production, the source catalog changes. If the catalog name is hardcoded in the SQL, you have to edit the source code or maintain separate versions of the pipeline.

That approach creates extra work. A developer might update the development pipeline but forget to make the same change in production. Over time, the two versions can behave differently even though they are supposed to perform the same job.

Databricks Lakeflow Pipelines now supports pipeline parameters that let you reuse the same SQL source code with different values. You can define defaults in pipeline settings and override them when starting an update or running a pipeline task in a job.

The feature became generally available on October 12, 2026. It is useful when a team needs to run the same data transformation against different environments, source tables, or date ranges.

What Are Lakeflow Pipeline Parameters?

A pipeline parameter is a key-value pair that supplies a value to your pipeline at runtime.

For example, a sales pipeline might use these parameters:

Parameter

Example value

Purpose

source_catalog

sales_dev

Select the catalog containing the source data

source_schema

transactions

Select the source schema

start_date

2026-01-01

Limit the data to a chosen start date

source_table

orders

Choose which table to read

These are example names and values. You define the parameters that match your own pipeline.

Instead of writing sales_dev directly into a SQL statement, you reference source_catalog. When you run the pipeline in production, you can provide sales_prod as the value without changing the SQL file.

Pipeline parameter values are always strings. Keys can contain letters, numbers, underscores, hyphens, and periods.

One important detail: named pipeline parameters are available in SQL source code only. If your pipeline uses Python source code, use the pipeline's Configuration field and read values with spark.conf.get() instead. The two approaches serve related purposes, but they use different syntax.

Define Parameters in Pipeline Settings

You can set default parameter values through the Databricks workspace UI, a pipeline JSON definition, or Declarative Automation Bundles.

To configure them through the UI:

  1. Open Jobs and Pipelines in your Databricks workspace.

  2. Select the pipeline you want to configure.

  3. Open Settings.

  4. Find the Parameters section and select Edit.

  5. Add the parameter keys and their default values.

  6. Save the settings.

For a pipeline managed through YAML, you can define the same defaults in its configuration.

resources:
  pipelines:
    sales_pipeline:
      name: Sales Summary Pipeline
      parameters:
        source_catalog: sales_dev
        source_schema: transactions
        start_date: "2026-01-01"

Here, the pipeline has three parameters. If an update runs without overrides, it uses these values.

Keeping the defaults in configuration makes the pipeline easier to deploy across environments. Developers can change the values without modifying the SQL transformation itself.

Reference Parameters in SQL

In SQL source code, reference a parameter by placing a colon before its name.

Consider a pipeline that creates a summary of sales by customer. Without parameters, the SQL might contain a hardcoded catalog name and date:

CREATE OR REFRESH MATERIALIZED VIEW customer_sales AS
SELECT
    customer_id,
    COUNT(order_id) AS order_count,
    SUM(total_amount) AS total_sales
FROM sales_dev.transactions.orders
WHERE order_date >= '2026-01-01'
GROUP BY customer_id;

This works for one environment, but changing the source catalog or start date requires editing the SQL.

With pipeline parameters, the query becomes:

CREATE OR REFRESH MATERIALIZED VIEW customer_sales AS
SELECT
    customer_id,
    COUNT(order_id) AS order_count,
    SUM(total_amount) AS total_sales
FROM :source_catalog.transactions.orders
WHERE order_date >= :start_date
GROUP BY customer_id;

The SQL stays the same when you switch environments. You only change the parameter values.

Databricks binds the parameter values as strings when the update runs. If a query uses a parameter in a catalog, schema, or table name, use IDENTIFIER() where required for the identifier position.

For example:

USE CATALOG IDENTIFIER(:source_catalog);
USE SCHEMA IDENTIFIER(:source_schema);

This is useful when you need to select a catalog or schema dynamically. Make sure the parameter value is a valid identifier for the object you want to access.

If the SQL references a parameter that has no value at runtime, the pipeline update fails. Extra parameters that the source code never references are ignored.

Override Parameters Without Editing the Pipeline

Default values are useful, but many pipelines need different values for different runs.

Imagine that your team runs a sales pipeline every day. Most runs process data from the production catalog, but a developer occasionally needs to test the same transformation against a development catalog.

You can override the parameters for one update without changing the saved defaults.

Run with different settings

Open the pipeline in the workspace and select Run with different settings. Change the values in the Parameters section and start the update.

The override applies to that update. The pipeline's saved defaults remain unchanged.

This is useful for testing a new source, checking a particular date range, or running a controlled backfill.

Pass parameters through Lakeflow Jobs

You can also supply parameters when a pipeline runs as a task in a Lakeflow Job. The job can provide values directly or use dynamic value references.

For example, a job parameter named region can be passed to a pipeline using a reference such as:

{{job.parameters.region}}

A job can also use values produced by an upstream task, such as:

{{tasks.prepare_data.values.source_catalog}}

This lets one task prepare a value and another task use it when running the pipeline.

Lakeflow Jobs automatically passes job parameters down to pipeline tasks. You do not have to declare every job parameter in pipeline settings just to make it available to SQL source code.

Understand Parameter Precedence

A parameter can be defined in more than one place. For example, a pipeline might have a default catalog, the parent job might specify another catalog, and a single job run might override it again.

Databricks resolves these values using the following precedence, from highest to lowest:

Priority

Source

Example

1

Job run parameters

A one-time override for a particular run

2

Job parameters

Defaults defined on the parent job

3

Pipeline task parameters

Values configured on the pipeline task

4

Pipeline parameters

Defaults saved in pipeline settings

The highest-priority value wins when the same key appears in multiple places.

This is worth checking when a pipeline unexpectedly reads from the wrong catalog or processes an unexpected date range. The saved pipeline default might be correct, but a job-level value could be overriding it.

Record the intended parameter values in your deployment process and check the run history when troubleshooting. Databricks records the parameters for individual updates and displays them in the pipeline run list.

Pipeline Parameters vs. the Configuration Field

Lakeflow pipelines also support the Configuration field, which exposes key-value pairs as Spark configuration values. It can still be used alongside pipeline parameters, but the two mechanisms are not interchangeable.

Feature

Pipeline parameters

Configuration field

Main purpose

Values that change between updates

Spark settings and configuration values

SQL access

Named syntax such as :source_catalog

${key} syntax

Python access

Named parameter syntax is not supported

Read values with spark.conf.get()

Job parameter pushdown

Supported

Uses its own configuration mechanism

Typical use

Source catalog, table, date range

Runtime configuration or values needed by Python code

For a SQL pipeline, use named parameters when the value should change between updates. For Python code, the Configuration field remains the supported way to pass values that the code reads through Spark configuration.

For example, a Python pipeline can read a configuration value like this:

from pyspark import pipelines as dp
from pyspark.sql.functions import col

@dp.table
def customer_events():
    start_date = spark.conf.get("mypipeline.start_date")

    return (
        spark.read.table("source_table")
        .where(col("event_date") > start_date)
    )

The mypipeline.start_date value must be configured in the pipeline's Configuration settings. This is not the same as declaring a named pipeline parameter and expecting Python to read it directly.

Common Mistakes to Avoid

Forgetting to provide a required value

If your SQL uses :source_catalog but no default or runtime override supplies that parameter, the update fails.

Check the parameter names in the SQL and compare them with the values defined in pipeline settings and the job configuration.

Expecting named parameters to work in Python

Named pipeline parameter syntax is for SQL source code. Python code should use the Configuration field and retrieve the value through spark.conf.get().

Assuming every date filter is incremental

Parameterizing a date filter does not guarantee that a materialized view can refresh incrementally.

For example, a predicate that places both a lower and an upper bound on a date range can invalidate incremental processing and trigger a full refresh on each update. If incremental refresh matters, test the query's refresh behavior before using a changing date range in production.

Running overlapping updates

A Lakeflow pipeline runs one update at a time. Jobs that attempt to run the same pipeline concurrently can have their concurrency capped at one in supported scenarios.

If you need to process many parameter combinations, design the job with this limitation in mind. Do not assume that several updates of the same pipeline will run in parallel.

Changing source values without checking permissions

Changing the catalog or schema parameter changes which objects the pipeline attempts to read. The identity used to run the pipeline must have access to the selected objects. A valid parameter value does not guarantee that the pipeline has permission to read the source.

Advantages and Disadvantages

Advantages

Disadvantages

Best Practices for Production Pipelines

Use meaningful parameter names. Names such as source_catalog, source_schema, and start_date are easier to understand than short names such as c1 or d1.

Keep defaults safe. A development pipeline should default to development data unless there is a clear reason otherwise. Avoid defaults that could accidentally direct testing work to production.

Review job overrides. When a pipeline is scheduled through Lakeflow Jobs, check the job and task parameters as well as the pipeline defaults. The higher-priority value takes precedence.

Test SQL with every supported value. Verify that the pipeline reads the intended tables and produces the expected results in each environment.

Check refresh behavior. If a materialized view must refresh incrementally, confirm that the parameterized query supports that behavior.

Use Configuration for Python. Do not mix up named SQL parameters with Spark configuration values. Keep the mechanism consistent with the source language.

Conclusion

Lakeflow pipeline parameters let Databricks developers reuse SQL source code across environments and datasets. You can define defaults in pipeline settings, override them for individual updates, and pass values through Lakeflow Jobs.

The feature reduces the need to maintain separate SQL files for development and production. To use it reliably, understand parameter precedence, provide every required value, and check how changes affect permissions and materialized-view refresh behavior.

For teams maintaining several pipelines that differ mainly in their source catalog, schema, table, or date range, parameters offer a straightforward way to reduce repeated code and simplify deployment.