Backing up Kubernetes workloads is easy when the amount of data is small. The problem starts when applications use large PersistentVolumes.
Imagine a database using a 1 TB volume. If only 5 GB of data changed since yesterday, copying the entire 1 TB volume again is wasteful. It takes more time, uses more network bandwidth, and requires more storage.
This is where Changed Block Tracking (CBT) becomes useful.
Changed Block Tracking allows a storage system to identify which blocks have changed between two snapshots. A backup application can then focus on those blocks instead of scanning and transferring the entire volume.
Kubernetes introduced support for this capability through CSI snapshot metadata APIs. The feature was initially introduced as Alpha and moved to Beta with the external-snapshot-metadata v1.0.0 release in March 2026. The current Beta implementation supports block volumes, not file volumes.
What Is Changed Block Tracking?
Changed Block Tracking is a way of answering a simple question:
Which blocks of this volume changed since the previous snapshot?
Traditional backup systems often have to read a large amount of data to determine what should be backed up.
With CBT, the storage system can provide metadata describing allocated blocks and changes between snapshots.
For example, assume a 500 GB volume contains:
A traditional full backup may process a large portion of the volume.
With CBT, the backup application can ask the storage system for the blocks that changed between two snapshots and transfer only the relevant data.
The important point is that CBT does not replace snapshots. It works with snapshots to make incremental backup operations more efficient.
Why Kubernetes Backups Need CBT
PersistentVolumes are commonly used by databases, file-processing applications, analytics systems, and other stateful workloads.
Consider a PostgreSQL database running on a 1 TB PersistentVolume.
The database may change only a small amount of data every day, but the volume itself remains 1 TB.
Without block-level change information, the backup application may need to process a much larger amount of data than the application actually changed.
That can affect:
Backup duration
Network traffic
Storage consumption
CPU usage
I/O pressure
Backup windows
CBT gives the backup application more precise information about what changed.
The Kubernetes project describes the feature as a way for CSI storage drivers to identify changed blocks between snapshots so backup applications can concentrate on changed data rather than processing the entire volume.
How CBT Fits Into Kubernetes Storage
CBT is not implemented entirely inside the Kubernetes API server.
It works through the Container Storage Interface (CSI) and several components work together.
A simplified flow looks like this:
Application
|
v
PersistentVolumeClaim
|
v
PersistentVolume
|
v
CSI Driver
|
v
Storage System
|
+---- Volume Snapshot
|
+---- Changed Block Metadata
|
v
Backup Application
The storage provider is responsible for knowing which blocks are allocated and which blocks changed.
The CSI layer exposes the required functionality.
The Kubernetes-facing metadata service makes that information available to backup applications.
The Three Important Components
The CBT implementation has three important pieces.
CSI Snapshot Metadata Service
The CSI driver can provide a snapshot metadata service.
This service exposes information about snapshot blocks through gRPC APIs.
Two important operations are:
GetMetadataAllocated
GetMetadataDelta
GetMetadataAllocated provides information about blocks allocated in a snapshot.
GetMetadataDelta provides information about blocks changed between two snapshots of the same volume.
This distinction is important for backup software because an initial backup and an incremental backup do not have the same requirements.
SnapshotMetadataService Custom Resource
Kubernetes needs a way to discover that a CSI driver provides snapshot metadata.
The SnapshotMetadataService Custom Resource is used for this purpose.
A simplified example looks like this:
apiVersion: cbt.storage.k8s.io/v1beta1
kind: SnapshotMetadataService
metadata:
name: example.csi.driver
spec:
address: snapshot-metadata.example.svc:8443
The actual resource contains additional information such as the service endpoint, CA certificate, and authentication audience.
The important idea is that the resource advertises the availability of the snapshot metadata service to clients.
External Snapshot Metadata Sidecar
The external-snapshot-metadata sidecar sits between Kubernetes clients and the CSI driver's snapshot metadata service.
A simplified architecture looks like this:
Backup Application
|
v
Kubernetes SnapshotMetadataService
|
v
External Snapshot Metadata Sidecar
|
v
CSI Snapshot Metadata Service
|
v
Storage System
The sidecar handles the Kubernetes-facing part of the communication and connects to the CSI driver's implementation.
The Kubernetes CSI project maintains this sidecar as part of the CBT implementation.
A Simple Backup Example
Let's consider a practical example.
Suppose an application uses a 100 GB block volume.
At 10:00 AM, the backup system creates Snapshot A.
Volume
|
+-- Snapshot A
The application continues running and changes some data.
At 10:00 AM the next day, the backup system creates Snapshot B.
Volume
|
+-- Snapshot A
|
+-- Snapshot B
Now the backup application needs to know:
What changed between Snapshot A and Snapshot B?
Instead of reading the complete volume, it can request delta metadata.
Conceptually:
GetMetadataDelta(
previousSnapshot = Snapshot A,
currentSnapshot = Snapshot B
)
The response represents the blocks that changed.
The backup application can then read and transfer those blocks.
Snapshot A
|
| compare
v
Snapshot B
|
v
Changed Blocks
|
v
Backup Storage
This is the basic idea behind incremental block-level backups.
Initial Backup vs Incremental Backup
CBT becomes easier to understand when you separate the first backup from later backups.
Initial Backup
For the first backup, there is no previous snapshot to compare against.
The backup application may request allocated block information.
Conceptually:
GetMetadataAllocated(Snapshot A)
The result tells the backup system which blocks are allocated.
The application can use that information to build the initial backup.
Incremental Backup
For a later backup, there is a previous snapshot.
For example:
Snapshot A -> Snapshot B
The backup application can request:
GetMetadataDelta(Snapshot A, Snapshot B)
The returned metadata identifies the changes between the two snapshots.
The backup application can then process those blocks instead of treating the entire volume as new data.
Why Block Tracking Is Different From File-Level Backup
A common assumption is that incremental backup means checking which files changed.
CBT works at a lower level.
Consider a database file:
database.db
The file may be several hundred gigabytes.
A small database update could change only a few blocks inside that file.
From a file-level perspective:
database.db = modified
From a block-level perspective:
Block 1000 = unchanged
Block 1001 = unchanged
Block 1002 = changed
Block 1003 = unchanged
Block 1004 = changed
CBT is interested in the second view.
This can be particularly useful for applications such as databases where a small logical change can occur inside a large file.
CBT Does Not Mean Every Backup Becomes Small
This is an important point.
CBT does not guarantee that every incremental backup will contain only a tiny amount of data.
If an application changes most of the volume, the delta can still be large.
For example:
Volume size: 1 TB
Changed data: 5 GB
An incremental backup may be relatively small.
But if the workload changes:
Volume size: 1 TB
Changed data: 700 GB
the backup still has a large amount of data to process.
CBT provides information about the changed blocks. It does not reduce the amount of data that actually changed.
Block Volume Support Matters
The current CBT API is designed for block volumes.
It does not provide the same changed-block tracking capability for file volumes or network file shares.
That means you should not assume that every PersistentVolume automatically benefits from CBT.
Before planning a backup architecture around this feature, check:
Whether your volume is a block volume.
Whether the CSI driver supports the required snapshot functionality.
Whether the CSI driver supports snapshot metadata.
Whether the backup application understands the CBT API.
Whether the required Kubernetes and CSI versions are supported.
The current Beta documentation lists Kubernetes 1.33 as the minimum Kubernetes version and CSI specification 1.10 or newer for this implementation.
What a Backup Application Actually Does
A backup application generally needs to perform more work than simply calling the metadata API.
A simplified workflow looks like this:
1. Create or identify a snapshot
2. Wait until the snapshot is ready
3. Identify the previous backup snapshot
4. Request changed-block metadata
5. Receive block metadata as a stream
6. Read the corresponding data
7. Transfer changed data
8. Store backup metadata
9. Record the snapshot relationship
The streaming part is important.
Large volumes can produce a large amount of metadata, so the API is designed to return metadata through streaming responses rather than requiring everything to be loaded into memory at once.
Why Streaming Matters
Imagine a large volume with millions of tracked blocks.
Returning every block in one huge response could create unnecessary memory pressure.
Instead, a backup application can process the metadata incrementally.
Conceptually:
CSI Driver
|
| block metadata
v
+---------+
| Chunk 1 |
+---------+
|
v
+---------+
| Chunk 2 |
+---------+
|
v
+---------+
| Chunk 3 |
+---------+
|
v
Backup Application
The backup application processes each portion and continues until the stream is complete.
This approach is much more suitable for large datasets.
CBT and Kubernetes VolumeSnapshots
CBT works alongside the existing Kubernetes volume snapshot model.
A VolumeSnapshot represents a point-in-time snapshot of a PersistentVolume through the CSI storage ecosystem.
For example:
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
name: database-snapshot
spec:
volumeSnapshotClassName: example-snapshot-class
source:
persistentVolumeClaimName: database-pvc
The snapshot itself is not the backup application.
It provides the point-in-time reference that can be used by the backup workflow.
CBT adds another piece of information:
Snapshot A
+
Snapshot B
|
v
Changed Blocks
This makes snapshots more useful as reference points for incremental backup systems.
Common Mistakes
Assuming Every CSI Driver Supports CBT
Installing Kubernetes 1.37 does not automatically make every storage driver capable of changed block tracking.
CBT requires support from the CSI/storage side.
Always verify the capabilities of the storage driver you are using.
Treating CBT as a Replacement for Backups
CBT is not a backup system.
It provides changed-block metadata that a backup application can use.
You still need:
Backup storage
Retention policies
Recovery procedures
Encryption
Access control
Monitoring
Backup validation
Ignoring Snapshot Relationships
Incremental backup depends on knowing which snapshots are being compared.
For example:
Backup 1 -> Snapshot A
Backup 2 -> Snapshot B
Backup 3 -> Snapshot C
The backup application must maintain the relationship between these points.
Losing the required metadata can make incremental recovery more complicated.
Assuming Changed Blocks Equal Changed Files
They are different concepts.
CBT works at the storage block level.
A database might modify a small section of a large database file. CBT can identify the affected blocks even though the entire file appears modified from a file-system perspective.
Forgetting Recovery Testing
A successful backup job does not automatically mean that the backup is recoverable.
A production backup process should periodically test restoration.
For example:
Production Volume
|
v
Snapshot A
|
v
Initial Backup
|
v
Snapshot B
|
v
Incremental Backup
|
v
Restore Test
|
v
Application Validation
The final step is important.
The backup is useful only if the application can actually recover from it.
Best Practices for Using CBT
Verify Storage Support First
Before designing the backup architecture, confirm that the CSI driver supports:
Do not assume that a Kubernetes cluster upgrade automatically provides storage-driver support.
Keep Snapshot Chains Manageable
Incremental backup systems often depend on relationships between snapshots.
Define a clear retention policy and make sure your backup application knows which snapshots are still required.
Monitor Backup Size
Track the amount of data transferred during each backup.
For example:
Backup #1 420 GB
Backup #2 7 GB
Backup #3 12 GB
Backup #4 9 GB
Backup #5 310 GB
A sudden increase can indicate a workload change or backup-chain behavior that deserves investigation.
Test Restore Regularly
Do not measure backup success only by whether the backup job finishes successfully.
Test:
Backup
|
v
Restore
|
v
Mount Volume
|
v
Start Application
|
v
Validate Data
This gives you much more confidence in the recovery process.
Protect Backup Metadata
The changed-block information is useful only when the backup system maintains the information needed to interpret the incremental chain.
Protect backup metadata with the same care as the backup data itself.
Advantages of Changed Block Tracking
Area | Traditional Full Processing | CBT-Based Incremental Processing |
|---|
Data processed | Large volume | Changed blocks |
Network usage | Higher | Potentially lower |
Backup window | Longer for large volumes | Potentially shorter |
Storage usage | More repeated data | Less repeated data |
Database workloads | Can be expensive | Better suited to small block changes |
Dependency | Backup system scans/processes volume | Requires CSI/storage support |
The actual improvement depends on how much data changes and how efficiently the storage driver and backup application implement the feature.
Limitations to Keep in Mind
CBT is useful, but it is not a universal solution.
The main limitations are:
It requires CSI/storage-driver support.
Current support focuses on block volumes.
Backup software must understand the metadata API.
Changed metadata can still be large for heavily modified volumes.
Snapshot and backup lifecycle management remains necessary.
Restore testing is still required.
Storage-provider implementation quality matters.
The feature is currently in Beta, so organizations should evaluate compatibility and operational behavior before making it a core dependency for critical backup workflows.
A Practical Way to Evaluate CBT
If you are considering CBT for a production environment, start with a non-critical workload.
Use a test volume and measure:
1. Full backup size
2. Incremental backup size
3. Backup duration
4. Network traffic
5. Storage I/O
6. Restore duration
7. Restore success rate
Then change different amounts of data and repeat the test.
For example:
Test 1: Change 1% of the volume
Test 2: Change 10% of the volume
Test 3: Change 50% of the volume
Test 4: Change 90% of the volume
This gives you a much better understanding of how CBT behaves with your particular storage system and workload.
Summary
Changed Block Tracking adds a useful capability to the Kubernetes storage ecosystem for incremental backups.
Instead of forcing a backup application to process an entire volume every time, CBT allows a compatible CSI storage system to report allocated blocks and the blocks that changed between snapshots.
The basic workflow is straightforward:
PersistentVolume
|
v
Snapshot A
|
| application changes data
v
Snapshot B
|
v
GetMetadataDelta
|
v
Changed Blocks
|
v
Incremental Backup
The important thing to remember is that CBT is not the backup itself. It is a mechanism that gives backup software better information about storage changes.
For large Kubernetes workloads, especially stateful applications, that information can help reduce unnecessary data processing and make incremental backup workflows more efficient.
Before using it in production, verify CSI driver support, Kubernetes compatibility, backup-tool support, snapshot management, and—most importantly—restore behavior.
Join the conversation! Your thoughts help the community grow.