Changing a Kubernetes API version is not always just a manifest change.
An object can be served through one API version while its actual representation in storage remains an older version. Kubernetes converts between API versions when clients read objects, but simply changing the preferred API version does not automatically rewrite every existing object in storage.
This becomes important when removing an old API version, changing a CustomResourceDefinition storage version, or rotating encryption-at-rest configuration.
Kubernetes 1.37 makes this easier with Storage Version Migration. The built-in StorageVersionMigration API and controller are now stable and enabled by default.
The important part for cluster operators is knowing what must happen before an old storage version can safely be removed.
What Is a Storage Version?
Kubernetes objects have an API representation and a storage representation.
For example, a client might interact with a resource through:
v1beta1
while the API server stores it using:
v1
Kubernetes handles the conversion between these representations.
The storage version is therefore not necessarily the same as the version used by your application or kubectl command.
A simplified flow looks like this:
Client
|
| v1beta1
v
API Server
|
| Convert
v
Storage Version
|
v
etcd
This abstraction is useful because clients can use different supported API versions without needing to know exactly how the object is serialized in storage.
Why Storage Versions Matter During API Upgrades
Suppose a CustomResourceDefinition currently supports:
v1alpha1
v1beta1
and v1beta1 is the current storage version.
You then introduce:
v1
and make v1 the new storage version.
New writes will use v1.
Existing objects do not automatically change just because the CRD's storage version changed.
You can therefore end up with:
Existing object A -> v1beta1
Existing object B -> v1beta1
Existing object C -> v1
New object D -> v1
This mixed state is allowed.
The problem appears when you want to remove v1beta1.
If objects are still stored using that version, removing the old version too early can leave Kubernetes unable to decode those objects correctly.
Changing the Storage Version Does Not Migrate Existing Objects
This is one of the most important points.
Changing:
spec:
versions:
- name: v1
storage: true
does not rewrite every existing object.
New or updated objects use the new storage version.
Objects that are never modified can remain stored using an older version.
For example:
Before change:
Object A -> v1beta1
Object B -> v1beta1
Object C -> v1beta1
After v1 becomes storage:
Object A -> v1beta1
Object B -> v1beta1
Object C -> v1beta1
New Object D -> v1
If Object A is later updated:
Object A -> v1
But Object B and Object C can remain on v1beta1.
That is why an explicit storage migration is needed.
What Kubernetes 1.37 Changes
Kubernetes 1.37 makes Storage Version Migration generally available.
The built-in API is:
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
The Kubernetes control plane includes a StorageVersionMigrator controller that watches these objects.
Instead of writing custom scripts to read and rewrite every resource, you can create a declarative migration request.
The basic architecture is:
StorageVersionMigration
|
v
StorageVersionMigrator
|
v
API Server
|
v
Existing Objects
|
v
New Storage Version
This makes migration easier to automate and monitor.
Example: Migrating a Custom Resource
Suppose you have:
crontabs.example.com
and the CRD now uses v1 as its storage version.
Create a migration object:
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
name: crontabs-migration
spec:
resource:
group: example.com
resource: crontabs
Apply it:
kubectl apply -f crontabs-migration.yaml
The built-in controller processes the existing objects and rewrites them using the current storage version.
Monitoring the Migration
Do not treat creation of the StorageVersionMigration object as proof that migration has finished.
Check its status:
kubectl get storageversionmigration \
crontabs-migration -o yaml
A successful migration reports a Succeeded condition.
You can also wait for completion:
kubectl wait \
--for=condition=Succeeded \
storageversionmigration.storagemigration.k8s.io/crontabs-migration
The migration status gives operators something that manual rewrite scripts often lack: a Kubernetes object representing the migration itself.
That makes it easier to monitor the operation and include it in an upgrade workflow.
Checking a CRD Before Removing an API Version
Before removing an old CRD version, inspect:
kubectl get crd crontabs.example.com -o yaml
Pay particular attention to:
status:
storedVersions:
- v1beta1
- v1
The storedVersions field records versions that have been used as storage versions.
If an old version is still present, do not assume it is safe to remove it.
First migrate the existing objects.
The desired result might be:
status:
storedVersions:
- v1
Only after confirming that the old storage version is no longer needed should you complete the removal process.
Why storedVersions Matters
Consider this CRD:
spec:
versions:
- name: v1
served: true
storage: true
- name: v1beta1
served: false
storage: false
It might look like v1beta1 has already been removed from active service.
But that does not necessarily mean old v1beta1 objects have disappeared from storage.
The CRD status can still contain:
status:
storedVersions:
- v1beta1
- v1
That is a signal that the old storage version still matters.
Storage migration addresses the actual stored objects. After successful migration, the old version can eventually be removed from storedVersions as part of the CRD version-removal process.
API Version Removal vs Storage Migration
These are related but different operations.
Operation | What it changes |
|---|
Change served | Whether clients can use an API version |
Change storage | Which version new writes use |
Storage migration | Rewrites existing objects |
Remove old API version | Removes API compatibility |
Update storedVersions | Records which versions have been used for storage |
A safe migration needs these steps in the correct order.
Changing all of them at once is risky.
A Safer CRD Migration Sequence
A practical migration can follow this sequence.
Step 1: Add the New Version
Add the new API version to the CRD.
For example:
spec:
versions:
- name: v1
served: true
storage: true
- name: v1beta1
served: true
storage: false
Step 2: Confirm New Writes Use the New Version
Create or update a test object.
The new storage version should now be v1.
Existing objects may still be stored using the previous version.
Step 3: Start Storage Version Migration
Create the migration object:
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
name: crontabs-migration
spec:
resource:
group: example.com
resource: crontabs
Step 4: Wait for Completion
kubectl wait \
--for=condition=Succeeded \
storageversionmigration.storagemigration.k8s.io/crontabs-migration
Step 5: Verify the CRD
Check:
kubectl get crd crontabs.example.com -o yaml
Confirm that the migration has completed and the stored-version state is what you expect.
Step 6: Remove the Old Version
Only after successful migration should you proceed with removing the old API version according to the CRD upgrade plan.
This sequence avoids creating a gap between API compatibility and the actual representation of existing objects.
Storage Migration and Encryption at Rest
Storage version migration is not limited to API version changes.
It can also be used when data needs to be rewritten because encryption-at-rest configuration changes.
For example, suppose Secrets were encrypted using:
key1
and the cluster rotates to:
key2
Existing Secrets do not automatically get rewritten merely because the encryption configuration changed.
The API server needs to read and write the objects again so that the new encryption configuration is applied.
A storage migration can be used for this purpose.
The migration object can target Secrets:
apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
name: secrets-migration
spec:
resource:
group: ""
resource: secrets
Apply it:
kubectl apply -f secrets-migration.yaml
Then monitor the migration:
kubectl wait \
--for=condition=Succeeded \
storageversionmigration.storagemigration.k8s.io/secrets-migration
The same principle applies: data must be actively rewritten for the new storage representation to take effect.
Built-In Resources and Custom Resources
Storage migration can apply to both built-in Kubernetes resources and Custom Resources.
For Custom Resources, the storage version is explicitly defined by the CRD.
For built-in resources, Kubernetes controls the storage representation as part of the API machinery.
This distinction matters when planning upgrades.
For example:
Custom Resource
|
`-- CRD defines storage version
Built-in Resource
|
`-- Kubernetes defines storage representation
Do not assume that every API migration requires exactly the same procedure.
What to Check Before a Kubernetes Upgrade
Storage version migration should be part of the upgrade review, not something discovered after the control plane has already been upgraded.
Before upgrading, inventory APIs that are changing.
Look for:
Removed API versions
Deprecated CRD versions
CRDs changing storage versions
APIs whose stored representation has changed
Encryption-at-rest changes
Controllers that still use old API versions
Admission webhooks that expect old schemas
A useful checklist is:
API inventory
|
v
Identify version changes
|
v
Check CRD storedVersions
|
v
Select new storage versions
|
v
Migrate existing objects
|
v
Verify migration
|
v
Remove old API versions
Common Mistakes
Removing an API Version Before Migrating Objects
This is the most serious mistake.
If objects remain stored using an old version and that version is removed prematurely, the API server may no longer be able to decode those objects correctly.
Assuming New Writes Migrate Old Objects
They do not.
Changing the storage version affects new writes and updates.
Objects that are never modified can remain on the previous storage version.
Checking Only the CRD Specification
Looking only at:
spec:
versions:
is not enough.
Inspect:
status:
storedVersions:
as part of the migration review.
Starting Migration and Immediately Removing the Old Version
Migration is asynchronous.
Wait for the Succeeded condition before continuing.
Ignoring Encryption Changes
Changing encryption configuration without rewriting existing objects can leave older data using the previous encryption configuration.
Include storage migration in encryption-key rotation procedures where appropriate.
Using Manual Rewrite Scripts Without a Reason
Manual kubectl get and kubectl replace loops can work, but they are easy to get wrong and harder to monitor.
Kubernetes 1.37 provides a built-in migration controller, so use it where it fits the workload.
Troubleshooting Migration Failures
Migration Does Not Complete
Inspect the migration object:
kubectl get storageversionmigration \
crontabs-migration -o yaml
Look at its conditions and status.
Then inspect the target resource and API server logs for errors.
Resource Is Not Migratable
Storage Version Migration requires the resource to have an integer resource version.
Standard Kubernetes resources and CRDs satisfy this requirement.
Aggregated APIs may not.
If the target resource does not meet the migration requirements, the migration can fail.
storedVersions Still Contains the Old Version
First confirm that the migration actually succeeded.
If it did, check whether the CRD changed during the migration.
A CRD modification during migration can require the migration to be retried before safely deprecating the old storage version.
Objects Still Appear in the Old API Version
Remember that the API version used by a client is not necessarily the storage version.
An object can still be requested through a served API version even after its underlying storage representation has changed.
Focus on the storage state, not only the version displayed by the client.
Best Practices
Treat API Removal as a Storage Migration Project
Do not plan an API removal as only a YAML update.
Include:
API compatibility
storage version
existing objects
CRD status
controllers
webhooks
migration
verification
Automate the Migration
For Kubernetes 1.37, use the declarative StorageVersionMigration API where applicable.
This gives the migration a Kubernetes resource that can be inspected and monitored.
Verify Before Removing Old Versions
Do not rely on assumptions.
Check migration status and the resulting CRD state before removing old versions.
Test With Real Object Data
A test cluster containing one simple object may not reveal problems caused by:
Use representative data during migration testing.
Include Controllers and Webhooks
An API migration is incomplete if an existing controller still depends on the removed API version.
Review all components that create, update, validate, or convert the resource.
Advantages and Disadvantages
Advantages
Kubernetes 1.37 provides a built-in storage migration API.
Migrations are declarative.
Progress can be monitored through Kubernetes status.
Supports migration of existing objects without manual rewrite scripts.
Useful for both API storage changes and encryption-at-rest rewrites.
Can be included in CRD upgrade workflows.
Disadvantages
Migration is still an operational task that needs verification.
Large resources can take time to rewrite.
Incorrect CRD conversion logic can cause migration problems.
Aggregated APIs may not satisfy migration requirements.
Removing an API version still requires careful dependency analysis.
A successful migration does not automatically update every part of an application's API compatibility.
A Practical Upgrade Checklist
Before removing or changing an API version, check:
Check | What to verify |
|---|
API inventory | Identify APIs being deprecated or removed |
CRDs | Review all affected versions |
Storage version | Identify the current and target versions |
storedVersions
| Check which versions have been used |
Controllers | Confirm they support the target API |
Webhooks | Confirm conversion and validation behavior |
Existing objects | Confirm they can be migrated |
Migration object | Create StorageVersionMigration |
Migration status | Wait for Succeeded |
CRD status | Confirm old storage version is no longer needed |
Encryption | Rewrite objects after encryption changes |
Final removal | Remove the old API only after verification |
Conclusion
Kubernetes API upgrades become much safer when storage versions are treated separately from the API versions clients use.
Changing a CRD's storage version affects new writes, but existing objects can remain stored using an older representation until they are rewritten. That is why removing an API version before migrating stored objects can create problems.
Kubernetes 1.37 makes this process easier by making Storage Version Migration stable and enabling it by default. The StorageVersionMigration API provides a declarative way to request migration, while the built-in controller performs the rewrite and reports progress.
Before removing an old API version, check the CRD's storedVersions, migrate existing objects, verify successful completion, and only then proceed with the API removal.
That extra step turns an API upgrade from a manifest change into a controlled storage migration, which is what Kubernetes actually needs when old representations are still present in the cluster.
Join the conversation! Your thoughts help the community grow.