If you've ever set up Auto Loader from a five-line tutorial snippet, it probably felt almost too easy. Point it at a folder, call .load(), watch a Bronze table fill up. Then, a few weeks into production, a source system quietly adds a field, a vendor overwrites a file instead of appending a new one, your landing zone crosses a few million objects, and someone on the data team asks why yesterday's numbers don't match what's in the table.
That gap — between "it works in the demo" and "it survives production" — is almost entirely about options you never had a reason to read the second time. This is a walkthrough of what those options actually do, not just what they're called.
What Auto Loader Actually Is
Strip away the marketing, and Auto Loader is a Structured Streaming source (cloudFiles) purpose-built for incrementally discovering and reading files that land in cloud object storage — S3, ADLS Gen2, or GCS. The alternative, a plain batch read like spark.read.json(path), has to enumerate every file in the directory on every run. That's fine at a few thousand files. It falls over at a few million, because listing cost and latency scale with the total size of the directory, not with what's new.
Auto Loader solves this by keeping durable state — which files it has already seen — so each run only has to deal with the delta.
# The plain-batch way: rescans everything, every time
df = spark.read.json("/mnt/landing/orders/")
# The Auto Loader way: remembers what it already processed
df = (
spark.readStream
.format("cloudFiles")
.option("cloudFiles.format", "json")
.load("/mnt/landing/orders/")
)
Everything past this point is really just: how does it discover files, what does it do with their schema, and how fast does it move.
The Shape of the API
There's one pattern to memorize; every option is layered on top of it.
# READ SIDE — Auto Loader as a streaming source
raw_df = (
spark.readStream
.format("cloudFiles") # tells Spark to use the Auto Loader source
.option("cloudFiles.format", "json") # tells Auto Loader what file type it's reading
.option("cloudFiles.schemaLocation", schema_path) # where inferred/evolved schema is persisted
.load(source_path)
)
# WRITE SIDE — a normal streaming sink, nothing Auto-Loader-specific here
(
raw_df.writeStream
.option("checkpointLocation", checkpoint_path) # required: tracks which files are already committed
.trigger(availableNow=True) # process the current backlog, then stop
.toTable("main.bronze.orders_raw")
)
Worth internalizing: options passed to .option() come from three unrelated systems, and Databricks doesn't distinguish between them syntactically:
Category | Example | Belongs to |
|---|
Auto Loader itself | cloudFiles.schemaEvolutionMode
| the cloudFiles source |
The underlying file-format reader | header, multiLine, dateFormat
| the CSV/JSON/etc. parser |
Structured Streaming | checkpointLocation, trigger
| Spark's streaming engine |
That's why cloudFiles.format and plain header can sit side by side in the same .option() chain. They're configuring completely different layers that happen to share one method call.
File Discovery: How Auto Loader Knows What's New
There are two genuinely different mechanisms here, and conflating them is one of the more common mistakes I see.
Directory listing is the default and the simplest to reason about: Auto Loader lists the source directory, diffs it against what it already recorded as processed, and picks up the rest.
list directory → compare against known state → process the difference
It requires zero setup, which makes it great for prototyping — and increasingly expensive as the directory grows, since every listing operation touches the whole tree, not just the new files.
File notification / event-based discovery flips the model: cloud storage tells Auto Loader when something new lands, instead of Auto Loader repeatedly asking.
.option("cloudFiles.useNotifications", "true") # classic mode: Auto Loader manages its own queue
versus the newer, generally preferred approach for Unity Catalog external locations:
.option("cloudFiles.useManagedFileEvents", "true") # Databricks manages shared event infrastructure for you
These two are not interchangeable, either in your config or in how you talk about them — classic notifications mean you (or Auto Loader on your behalf) provision a per-stream cloud queue; managed file events let multiple streams share event infrastructure attached to one external location, without you touching a queue at all.
One nuance worth remembering if you want to actually understand this rather than recite it: switching to event-based discovery doesn't mean directory listing disappears forever. Auto Loader can still fall back to a listing pass when a stream initializes for the first time, or when it needs to resynchronize after a long gap. "Event-driven" describes normal operation, not an absolute guarantee.
Schema Inference: The Default That Surprises Everyone
Feed Auto Loader this JSON:
{"customer_id": 501, "total": 129.99, "placed_at": "2026-09-20T14:00:00Z"}
You'd reasonably expect customer_id to come back as a long and total as a double. By default, for text-based formats (JSON, CSV, XML), it doesn't:
root
|-- customer_id: string
|-- total: string
|-- placed_at: string
|-- _rescued_data: string
Everything is a string. This isn't a bug — it's a deliberate hedge against schema evolution headaches. If one file has customer_id as 501 and the next has it as "501-A", treating both as strings means the stream doesn't choke on a type conflict it can't cleanly resolve.
To get real types, opt in explicitly:
.option("cloudFiles.inferColumnTypes", "true")
This recovers numerics and booleans, but it won't guess that a string looks like a timestamp — placed_at stays a string either way. For that, tell Auto Loader directly with schema hints:
.option("cloudFiles.schemaHints",
"total DOUBLE, customer_id LONG, placed_at TIMESTAMP")
Hints reach into nested structures too (shipping.zip STRING), and they're the cleanest way to pin down the columns you actually depend on downstream while leaving the long tail of fields to infer themselves.
Schema Evolution: Five Ways to Handle Drift
Say your source starts sending an extra field one day — loyalty_tier shows up where it never did before. Auto Loader gives you five distinct policies for that moment, and picking the wrong one is how people either lose data silently or get paged at 2 a.m. for a stream that "just stopped."
addNewColumns (the default). The first batch containing the new field fails the stream on purpose — but before it does, it writes the updated schema to cloudFiles.schemaLocation. Restart the stream and it picks the new column up cleanly, backfilling null for older rows that predate it.
.option("cloudFiles.schemaEvolutionMode", "addNewColumns")
The failure looks alarming the first time you see it, but it's really just Auto Loader forcing you to notice a schema change instead of silently absorbing it.
rescue. The table schema never changes. Anything unexpected — new fields, type mismatches, even case-mismatched field names — gets shoved into a JSON blob column (_rescued_data by default, renameable via rescuedDataColumn) along with the source file path.
.option("cloudFiles.schemaEvolutionMode", "rescue")
customer_id | total | _rescued_data
501 | 129.99 | null
502 | 88.50 | {"loyalty_tier":"GOLD","_file_path":"abfss://.../orders_0912.json"}
Nothing is lost, nothing halts. This is the mode I reach for whenever the upstream system is genuinely unpredictable and I'd rather quarantine surprises than stop the pipeline over them.
failOnNewColumns. New field appears, stream stops, and — unlike addNewColumns — restarting doesn't fix it. A human has to explicitly update the schema or supply a hint. Good for strict, contractual sources (think regulated data feeds) where an unannounced field is supposed to be treated as an incident, not routine drift.
none. New fields are simply dropped, with no failure and no evolution. If you haven't configured a rescue column, that data disappears without a trace — genuinely the one mode where you can lose information and never know it happened. Pair it with rescuedDataColumn unless you have a specific reason not to.
addNewColumnsWithTypeWidening (newer runtimes, public preview on DBR 16.4+). Same as addNewColumns, but it also safely widens compatible types — int to long, float to double — in place, without a full table rewrite, when a later file's values overflow the originally inferred type.
Throughput Controls
Two options govern batch size, and they interact in a way that trips people up in interviews as much as in production.
.option("cloudFiles.maxFilesPerTrigger", 1000) # default; hard cap on file count per micro-batch
.option("cloudFiles.maxBytesPerTrigger", "10g") # soft cap on data volume per micro-batch
The byte limit is a soft limit because Auto Loader never splits a single file across batches. If your limit is 10 GB and four 3 GB files are all sitting in the queue, the batch will pull in all four — 12 GB — rather than stop at three. Size your cluster around "limit plus one large file," not around the limit as a hard ceiling. Set both together and Auto Loader stops at whichever threshold it hits first — useful protection when file sizes vary wildly and a file-count cap alone wouldn't catch a batch of unusually large files.
Partitions, Filtering, and File Lifecycle
If your landing zone follows a Hive-style layout —
/landing/orders/region=EU/year=2026/month=09/orders.json
— telling Auto Loader about it lifts those path segments into real columns instead of leaving you to regex _metadata.file_path later:
.option("cloudFiles.partitionColumns", "region,year,month")
To restrict which files even get considered — say a shared landing zone with both orders_* and customers_* files — use a glob filter:
.option("pathGlobFilter", "orders_*.json")
A few lifecycle options round this out. cloudFiles.allowOverwrites (default false) controls whether a file that gets overwritten in place is eligible for reprocessing — leave it off unless your source genuinely rewrites files, and even then, handle deduplication explicitly downstream rather than depending on this flag alone. cloudFiles.maxFileAge doesn't limit which files get read by date, as the name might suggest — it controls how long Auto Loader retains file-tracking state for deduplication purposes; shrinking it trims memory usage at the risk of reprocessing or missing late arrivals. And cloudFiles.cleanSource (DBR 16.4+) lets Auto Loader move or delete files after they're successfully committed — genuinely useful once a landing zone has accumulated millions of files that will never be read again, but start with MOVE to an archive path and a generous retention window before ever reaching for DELETE.
Metadata: The Cheapest Debugging Insurance You'll Ever Add
Auto Loader exposes a hidden _metadata struct on every row — file path, name, size, and modification time — and pulling it into Bronze costs almost nothing:
enriched_df = (
raw_df.selectExpr(
"*",
"_metadata.file_path AS source_file_path",
"_metadata.file_modification_time AS source_file_modified_at"
)
)
The payoff shows up the first time someone asks "which file produced this bad row" and you can answer with a query instead of a multi-hour archaeology dig. I'd go as far as saying: if a Bronze table doesn't carry source_file_path and an ingestion timestamp, it isn't finished yet.
cloud_files_state(): The Audit Table You Already Have
This is genuinely underused. Every Auto Loader stream's checkpoint can be queried directly for its file-level history:
SELECT path, discovery_time, processed_time, ingestion_state
FROM cloud_files_state('/Volumes/main/bronze/checkpoints/orders')
WHERE processed_time IS NULL;
That single query answers "why hasn't file X shown up yet" without touching a log. Swap the WHERE clause and you can just as easily measure discovery-to-commit latency or confirm whether yesterday's files actually landed.
Triggers: availableNow vs. processingTime
For scheduled, incremental jobs — the overwhelming majority of Auto Loader use cases — Trigger.AvailableNow is the pattern to reach for:
(
enriched_df.writeStream
.option("checkpointLocation", checkpoint_path)
.trigger(availableNow=True)
.toTable("main.bronze.orders_raw")
)
It processes everything currently sitting in the backlog, across as many micro-batches as it needs (respecting your rate limits along the way), and then releases the cluster. It replaced the older Trigger.Once, which tried to cram the entire backlog into a single batch — a great way to run out of memory on a large first load.
processingTime keeps the cluster running continuously, checking for new files on a fixed interval:
.trigger(processingTime="30 seconds")
That buys you lower latency at the cost of a cluster that never spins down. Reach for it when near-real-time genuinely matters to the business, not because a continuously running stream sounds more sophisticated on an architecture diagram.
Exactly-Once, and What It Doesn't Promise
The guarantee comes from three things working together — file-level state tracking, the checkpoint, and Delta Lake's transactional writes. Remove any one and the guarantee goes with it.
What it deliberately does not promise is ordering. Auto Loader will not guarantee it discovers or processes files in the order they landed. So logic like "the last file processed reflects current state" is a bug waiting to happen. The safe pattern derives ordering from a timestamp or version column inside the data itself:
from pyspark.sql import Window
from pyspark.sql.functions import col, row_number
dedup_window = Window.partitionBy("customer_id").orderBy(col("event_time").desc())
latest_state = (
silver_df
.withColumn("rn", row_number().over(dedup_window))
.filter("rn = 1")
.drop("rn")
)
Never assume file arrival order equals event order — they aren't the same thing, and Auto Loader isn't promising they are.
Quick Reference
Concern | Option | Default |
|---|
File format | cloudFiles.format
| required |
Schema state location | cloudFiles.schemaLocation
| required for evolution/inference |
Process pre-existing files on first run | cloudFiles.includeExistingFiles
| true
|
Infer real types vs. strings | cloudFiles.inferColumnTypes
| false
|
Override inferred types | cloudFiles.schemaHints
| — |
Drift behavior | cloudFiles.schemaEvolutionMode
| addNewColumns
|
Files per micro-batch | cloudFiles.maxFilesPerTrigger
| 1000
|
Bytes per micro-batch (soft) | cloudFiles.maxBytesPerTrigger
| — |
Event-based discovery | cloudFiles.useManagedFileEvents
| false
|
Reprocess overwritten files | cloudFiles.allowOverwrites
| false
|
Post-ingestion cleanup | cloudFiles.cleanSource
| OFF
|
Extract path partitions | cloudFiles.partitionColumns
| — |
Join the conversation! Your thoughts help the community grow.