Kubernetes Storage Version Migration: What to Check Before Upgrading APIs

Changing a Kubernetes API version is not always just a manifest change.

An object can be served through one API version while its actual representation in storage remains an older version. Kubernetes converts between API versions when clients read objects, but simply changing the preferred API version does not automatically rewrite every existing object in storage.

This becomes important when removing an old API version, changing a CustomResourceDefinition storage version, or rotating encryption-at-rest configuration.

Kubernetes 1.37 makes this easier with Storage Version Migration. The built-in StorageVersionMigration API and controller are now stable and enabled by default.

The important part for cluster operators is knowing what must happen before an old storage version can safely be removed.

What Is a Storage Version?

Kubernetes objects have an API representation and a storage representation.

For example, a client might interact with a resource through:

v1beta1

while the API server stores it using:

v1

Kubernetes handles the conversion between these representations.

The storage version is therefore not necessarily the same as the version used by your application or kubectl command.

A simplified flow looks like this:

Client
  |
  | v1beta1
  v
API Server
  |
  | Convert
  v
Storage Version
  |
  v
etcd

This abstraction is useful because clients can use different supported API versions without needing to know exactly how the object is serialized in storage.

Why Storage Versions Matter During API Upgrades

Suppose a CustomResourceDefinition currently supports:

v1alpha1
v1beta1

and v1beta1 is the current storage version.

You then introduce:

v1

and make v1 the new storage version.

New writes will use v1.

Existing objects do not automatically change just because the CRD's storage version changed.

You can therefore end up with:

Existing object A -> v1beta1
Existing object B -> v1beta1
Existing object C -> v1
New object D      -> v1

This mixed state is allowed.

The problem appears when you want to remove v1beta1.

If objects are still stored using that version, removing the old version too early can leave Kubernetes unable to decode those objects correctly.

Changing the Storage Version Does Not Migrate Existing Objects

This is one of the most important points.

Changing:

spec:
  versions:
    - name: v1
      storage: true

does not rewrite every existing object.

New or updated objects use the new storage version.

Objects that are never modified can remain stored using an older version.

For example:

Before change:

Object A -> v1beta1
Object B -> v1beta1
Object C -> v1beta1

After v1 becomes storage:

Object A -> v1beta1
Object B -> v1beta1
Object C -> v1beta1
New Object D -> v1

If Object A is later updated:

Object A -> v1

But Object B and Object C can remain on v1beta1.

That is why an explicit storage migration is needed.

What Kubernetes 1.37 Changes

Kubernetes 1.37 makes Storage Version Migration generally available.

The built-in API is:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration

The Kubernetes control plane includes a StorageVersionMigrator controller that watches these objects.

Instead of writing custom scripts to read and rewrite every resource, you can create a declarative migration request.

The basic architecture is:

StorageVersionMigration
          |
          v
StorageVersionMigrator
          |
          v
API Server
          |
          v
Existing Objects
          |
          v
New Storage Version

This makes migration easier to automate and monitor.

Example: Migrating a Custom Resource

Suppose you have:

crontabs.example.com

and the CRD now uses v1 as its storage version.

Create a migration object:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: crontabs-migration
spec:
  resource:
    group: example.com
    resource: crontabs

Apply it:

kubectl apply -f crontabs-migration.yaml

The built-in controller processes the existing objects and rewrites them using the current storage version.

Monitoring the Migration

Do not treat creation of the StorageVersionMigration object as proof that migration has finished.

Check its status:

kubectl get storageversionmigration \
  crontabs-migration -o yaml

A successful migration reports a Succeeded condition.

You can also wait for completion:

kubectl wait \
  --for=condition=Succeeded \
  storageversionmigration.storagemigration.k8s.io/crontabs-migration

The migration status gives operators something that manual rewrite scripts often lack: a Kubernetes object representing the migration itself.

That makes it easier to monitor the operation and include it in an upgrade workflow.

Checking a CRD Before Removing an API Version

Before removing an old CRD version, inspect:

kubectl get crd crontabs.example.com -o yaml

Pay particular attention to:

status:
  storedVersions:
    - v1beta1
    - v1

The storedVersions field records versions that have been used as storage versions.

If an old version is still present, do not assume it is safe to remove it.

First migrate the existing objects.

The desired result might be:

status:
  storedVersions:
    - v1

Only after confirming that the old storage version is no longer needed should you complete the removal process.

Why storedVersions Matters

Consider this CRD:

spec:
  versions:
    - name: v1
      served: true
      storage: true
    - name: v1beta1
      served: false
      storage: false

It might look like v1beta1 has already been removed from active service.

But that does not necessarily mean old v1beta1 objects have disappeared from storage.

The CRD status can still contain:

status:
  storedVersions:
    - v1beta1
    - v1

That is a signal that the old storage version still matters.

Storage migration addresses the actual stored objects. After successful migration, the old version can eventually be removed from storedVersions as part of the CRD version-removal process.

API Version Removal vs Storage Migration

These are related but different operations.

A safe migration needs these steps in the correct order.

Changing all of them at once is risky.

A Safer CRD Migration Sequence

A practical migration can follow this sequence.

Step 1: Add the New Version

Add the new API version to the CRD.

For example:

spec:
  versions:
    - name: v1
      served: true
      storage: true
    - name: v1beta1
      served: true
      storage: false

Step 2: Confirm New Writes Use the New Version

Create or update a test object.

The new storage version should now be v1.

Existing objects may still be stored using the previous version.

Step 3: Start Storage Version Migration

Create the migration object:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: crontabs-migration
spec:
  resource:
    group: example.com
    resource: crontabs

Step 4: Wait for Completion

kubectl wait \
  --for=condition=Succeeded \
  storageversionmigration.storagemigration.k8s.io/crontabs-migration

Step 5: Verify the CRD

Check:

kubectl get crd crontabs.example.com -o yaml

Confirm that the migration has completed and the stored-version state is what you expect.

Step 6: Remove the Old Version

Only after successful migration should you proceed with removing the old API version according to the CRD upgrade plan.

This sequence avoids creating a gap between API compatibility and the actual representation of existing objects.

Storage Migration and Encryption at Rest

Storage version migration is not limited to API version changes.

It can also be used when data needs to be rewritten because encryption-at-rest configuration changes.

For example, suppose Secrets were encrypted using:

key1

and the cluster rotates to:

key2

Existing Secrets do not automatically get rewritten merely because the encryption configuration changed.

The API server needs to read and write the objects again so that the new encryption configuration is applied.

A storage migration can be used for this purpose.

The migration object can target Secrets:

apiVersion: storagemigration.k8s.io/v1
kind: StorageVersionMigration
metadata:
  name: secrets-migration
spec:
  resource:
    group: ""
    resource: secrets

Apply it:

kubectl apply -f secrets-migration.yaml

Then monitor the migration:

kubectl wait \
  --for=condition=Succeeded \
  storageversionmigration.storagemigration.k8s.io/secrets-migration

The same principle applies: data must be actively rewritten for the new storage representation to take effect.

Built-In Resources and Custom Resources

Storage migration can apply to both built-in Kubernetes resources and Custom Resources.

For Custom Resources, the storage version is explicitly defined by the CRD.

For built-in resources, Kubernetes controls the storage representation as part of the API machinery.

This distinction matters when planning upgrades.

For example:

Custom Resource
    |
    `-- CRD defines storage version

Built-in Resource
    |
    `-- Kubernetes defines storage representation

Do not assume that every API migration requires exactly the same procedure.

What to Check Before a Kubernetes Upgrade

Storage version migration should be part of the upgrade review, not something discovered after the control plane has already been upgraded.

Before upgrading, inventory APIs that are changing.

Look for:

• Removed API versions

• Deprecated CRD versions

• CRDs changing storage versions

• APIs whose stored representation has changed

• Encryption-at-rest changes

• Controllers that still use old API versions

• Admission webhooks that expect old schemas

A useful checklist is:

API inventory
    |
    v
Identify version changes
    |
    v
Check CRD storedVersions
    |
    v
Select new storage versions
    |
    v
Migrate existing objects
    |
    v
Verify migration
    |
    v
Remove old API versions

Common Mistakes

Removing an API Version Before Migrating Objects

This is the most serious mistake.

If objects remain stored using an old version and that version is removed prematurely, the API server may no longer be able to decode those objects correctly.

Assuming New Writes Migrate Old Objects

They do not.

Changing the storage version affects new writes and updates.

Objects that are never modified can remain on the previous storage version.

Checking Only the CRD Specification

Looking only at:

spec:
  versions:

is not enough.

Inspect:

status:
  storedVersions:

as part of the migration review.

Starting Migration and Immediately Removing the Old Version

Migration is asynchronous.

Wait for the Succeeded condition before continuing.

Ignoring Encryption Changes

Changing encryption configuration without rewriting existing objects can leave older data using the previous encryption configuration.

Include storage migration in encryption-key rotation procedures where appropriate.

Using Manual Rewrite Scripts Without a Reason

Manual kubectl get and kubectl replace loops can work, but they are easy to get wrong and harder to monitor.

Kubernetes 1.37 provides a built-in migration controller, so use it where it fits the workload.

Troubleshooting Migration Failures

Migration Does Not Complete

Inspect the migration object:

kubectl get storageversionmigration \
  crontabs-migration -o yaml

Look at its conditions and status.

Then inspect the target resource and API server logs for errors.

Resource Is Not Migratable

Storage Version Migration requires the resource to have an integer resource version.

Standard Kubernetes resources and CRDs satisfy this requirement.

Aggregated APIs may not.

If the target resource does not meet the migration requirements, the migration can fail.

storedVersions Still Contains the Old Version

First confirm that the migration actually succeeded.

If it did, check whether the CRD changed during the migration.

A CRD modification during migration can require the migration to be retried before safely deprecating the old storage version.

Objects Still Appear in the Old API Version

Remember that the API version used by a client is not necessarily the storage version.

An object can still be requested through a served API version even after its underlying storage representation has changed.

Focus on the storage state, not only the version displayed by the client.

Best Practices

Treat API Removal as a Storage Migration Project

Do not plan an API removal as only a YAML update.

Include:

API compatibility
storage version
existing objects
CRD status
controllers
webhooks
migration
verification

Automate the Migration

For Kubernetes 1.37, use the declarative StorageVersionMigration API where applicable.

This gives the migration a Kubernetes resource that can be inspected and monitored.

Verify Before Removing Old Versions

Do not rely on assumptions.

Check migration status and the resulting CRD state before removing old versions.

Test With Real Object Data

A test cluster containing one simple object may not reveal problems caused by:

• Large object counts

• Large object sizes

• Invalid historical data

• Older schema versions

• Conversion webhook behavior

Use representative data during migration testing.

Include Controllers and Webhooks

An API migration is incomplete if an existing controller still depends on the removed API version.

Review all components that create, update, validate, or convert the resource.

Advantages and Disadvantages

Advantages

• Kubernetes 1.37 provides a built-in storage migration API.

• Migrations are declarative.

• Progress can be monitored through Kubernetes status.

• Supports migration of existing objects without manual rewrite scripts.

• Useful for both API storage changes and encryption-at-rest rewrites.

• Can be included in CRD upgrade workflows.

Disadvantages

• Migration is still an operational task that needs verification.

• Large resources can take time to rewrite.

• Incorrect CRD conversion logic can cause migration problems.

• Aggregated APIs may not satisfy migration requirements.

• Removing an API version still requires careful dependency analysis.

• A successful migration does not automatically update every part of an application's API compatibility.

A Practical Upgrade Checklist

Before removing or changing an API version, check:

Conclusion

Kubernetes API upgrades become much safer when storage versions are treated separately from the API versions clients use.

Changing a CRD's storage version affects new writes, but existing objects can remain stored using an older representation until they are rewritten. That is why removing an API version before migrating stored objects can create problems.

Kubernetes 1.37 makes this process easier by making Storage Version Migration stable and enabling it by default. The StorageVersionMigration API provides a declarative way to request migration, while the built-in controller performs the rewrite and reports progress.

Before removing an old API version, check the CRD's storedVersions, migrate existing objects, verify successful completion, and only then proceed with the API removal.

That extra step turns an API upgrade from a manifest change into a controlled storage migration, which is what Kubernetes actually needs when old representations are still present in the cluster.