A Kubernetes application can continue running successfully after a configuration change, yet some of its Pods may still use the previous settings while others use the updated values.
This situation is easy to overlook because changing a ConfigMap or Secret does not always trigger a new Deployment rollout.
Consider an ASP.NET Core application running with three replicas. The application reads its database connection string, logging settings, and external service URLs from Kubernetes configuration.
A developer updates the database connection string in a ConfigMap or Secret. The Kubernetes object is updated successfully, but the existing containers continue using the environment variables they received when they started.
Later, Kubernetes creates another Pod because of scaling or a node failure. That new Pod receives the updated configuration, while the existing Pods continue using the old values.
The application is now running with different configurations across its replicas.
This can lead to inconsistent behavior, failed requests, database connection problems, and difficult production debugging.
The important point is that Kubernetes manages configuration objects and application Pods separately. Updating one does not automatically mean that every running application instance has started using the new configuration.
Understanding how ConfigMaps, Secrets, Pod templates, and rolling updates work together helps prevent these inconsistencies.
Understand Why Configuration Changes Do Not Always Restart Pods
Kubernetes Deployments create Pods from a Pod template.
When the Pod template changes, Kubernetes creates a new ReplicaSet and gradually replaces the existing Pods.
However, changing a ConfigMap or Secret referenced by a Deployment does not automatically change the Deployment's Pod template.
For example, consider this ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-config
data:
ASPNETCORE_ENVIRONMENT: Production
LoggingLevel: Information
PaymentServiceUrl: https://payments.example.com
The Deployment consumes these values as environment variables:
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
replicas: 3
selector:
matchLabels:
app: order-api
template:
metadata:
labels:
app: order-api
spec:
containers:
- name: order-api
image: registry.example.com/order-api:v1
envFrom:
- configMapRef:
name: order-api-config
Suppose a developer changes PaymentServiceUrl in the ConfigMap.
kubectl edit configmap order-api-config
The ConfigMap is updated, but the environment variables inside already-running containers remain unchanged.
Kubernetes does not recreate the Pods just because the referenced ConfigMap data changed.
A newly created Pod, however, receives the updated values.
This is why a Deployment can end up with different configuration values across its replicas.
A Deployment rollout is triggered by changes to its Pod template, not simply by changes to external configuration objects.
Understand How ConfigMaps Behave When Updated
ConfigMaps can provide configuration to containers in different ways.
The update behavior depends on how the application consumes the configuration.
Environment variables
When a ConfigMap is used to populate environment variables, the values are provided when the container starts.
Changing the ConfigMap does not update the environment variables of an existing process.
For example:
env:
- name: LoggingLevel
valueFrom:
configMapKeyRef:
name: order-api-config
key: LoggingLevel
If LoggingLevel changes from Information to Debug, the existing application process continues using the value it received at startup.
A new Pod must be created to receive the updated environment variable.
Mounted configuration files
A ConfigMap can also be mounted as a volume.
volumes:
- name: application-config
configMap:
name: order-api-config
containers:
- name: order-api
image: registry.example.com/order-api:v1
volumeMounts:
- name: application-config
mountPath: /app/config
readOnly: true
When a mounted ConfigMap changes, Kubernetes eventually updates the projected files.
However, this does not automatically mean that the application has loaded the new values.
The application must be designed to detect and reload configuration changes.
There is another important limitation: ConfigMap and Secret files mounted using subPath do not receive automatic updates.
For example:
volumeMounts:
- name: application-config
mountPath: /app/config/appsettings.json
subPath: appsettings.json
If the underlying ConfigMap changes, the application should not expect this subPath-mounted file to be updated automatically.
The same general distinction applies to Secrets: environment variables are fixed for the lifetime of the container process, while ordinary volume projections can be updated eventually.
Choose the configuration delivery method according to whether the application supports dynamic configuration reload or requires a restart.
Use a Rollout Restart for Simple Configuration Changes
One straightforward way to apply updated environment variables is to restart the Deployment after changing its ConfigMap or Secret.
For example:
kubectl apply -f order-api-config.yaml
Then trigger a restart:
kubectl rollout restart deployment/order-api
Kubernetes updates the Pod template with a restart annotation, causing the Deployment to create a new ReplicaSet and replace its Pods.
Monitor the rollout:
kubectl rollout status deployment/order-api
Check the available replicas:
kubectl get deployment order-api
And inspect the Pods:
kubectl get pods -l app=order-api
This approach works well for straightforward operational updates.
However, there is an important consideration when multiple configuration changes are deployed together.
If the ConfigMap is updated first and the Deployment restart happens later, there is a window in which new Pods created for other reasons may receive the updated configuration before the planned rollout begins.
For critical production changes, it is better to make configuration versioning and Pod replacement part of the same deployment process.
Trigger Rollouts Automatically When Configuration Changes
A more reliable approach is to make configuration changes part of the Pod template.
One common technique is to add a checksum annotation to the Pod template.
The deployment tool calculates a hash of the configuration file and places it in the Pod template metadata.
For example:
apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
replicas: 3
selector:
matchLabels:
app: order-api
template:
metadata:
labels:
app: order-api
annotations:
checksum/config: "replace-with-generated-checksum"
spec:
containers:
- name: order-api
image: registry.example.com/order-api:v1
envFrom:
- configMapRef:
name: order-api-config
The checksum value must be generated from the actual configuration content by the deployment process.
When the configuration changes, the checksum changes as well.
Because the annotation belongs to the Pod template, Kubernetes detects a template change and starts a new rollout.
The same approach can be used for configuration files and Secret references, provided the checksum is generated securely.
Do not place passwords, tokens, or other secret values directly in annotations. A checksum of the Secret content is sufficient to detect a change, but even hashes of low-entropy secrets can reveal information through guessing attacks. Use a protected deployment process and avoid exposing sensitive values in rendered manifests, logs, or public build artifacts.
With Helm, this pattern can be implemented using a template checksum:
spec:
template:
metadata:
annotations:
checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}
The template calculates the hash of the rendered ConfigMap.
When the ConfigMap content changes, Helm renders a different annotation value. Kubernetes then creates a new ReplicaSet.
The important requirement is that the checksum must be part of the Pod template, not only the Deployment's top-level metadata.
This approach is useful when configuration changes should always trigger a controlled application restart.
Use Versioned ConfigMaps for More Predictable Releases
Another approach is to create a new ConfigMap whenever the configuration changes instead of repeatedly modifying the same object.
For example, the first release uses:
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-config-v1
data:
LoggingLevel: Information
PaymentServiceUrl: https://payments.example.com
The Deployment references that version:
envFrom:
- configMapRef:
name: order-api-config-v1
For the next release, create another ConfigMap:
apiVersion: v1
kind: ConfigMap
metadata:
name: order-api-config-v2
data:
LoggingLevel: Debug
PaymentServiceUrl: https://payments.example.com
Update the Deployment reference:
envFrom:
- configMapRef:
name: order-api-config-v2
This changes the Pod template because the referenced ConfigMap name has changed.
Kubernetes creates a new ReplicaSet and gradually replaces the old Pods.
Versioned configuration provides several advantages:
Each application release references a specific configuration version.
Configuration changes are easier to review.
Previous configurations can be retained for recovery.
Deployment history becomes easier to understand.
Configuration changes are less likely to affect existing application instances unexpectedly.
This pattern is particularly useful when applications require configuration to remain consistent for their entire lifetime.
However, versioned ConfigMaps require a cleanup strategy. Old configuration objects should not be deleted while existing Pods, rollback revisions, or other workloads still depend on them.
Handle Secret Changes Without Exposing Sensitive Values
Secrets require additional care because they contain sensitive information such as database passwords, API keys, certificates, and authentication tokens.
Consider a Deployment that loads a database password from a Secret:
env:
- name: DatabasePassword
valueFrom:
secretKeyRef:
name: order-api-secrets
key: database-password
If the Secret is updated, the environment variable inside the running application does not change.
The application continues using the original value until its container is replaced.
A common production approach is to create a new Secret version and update the Deployment reference.
For example:
env:
- name: DatabasePassword
valueFrom:
secretKeyRef:
name: order-api-secrets-v2
key: database-password
This changes the Pod template and triggers a rollout.
When rotating database credentials, consider the order of operations carefully.
If the database immediately invalidates the old password, existing Pods may lose their database connections before the replacement Pods are ready.
A safer rotation process is:
Introduce the new credential while keeping the old credential temporarily valid.
Update the Kubernetes Secret or create a new version.
Deploy new Pods using the new credential.
Verify that all new Pods can connect successfully.
Confirm that the old Pods have terminated.
Revoke the old credential after the transition is complete.
The exact process depends on the database or external service.
The objective is to avoid a situation in which neither the old nor the new application instances can authenticate successfully.
Never print Secret values during troubleshooting or include them in application deployment logs.
Configure Rolling Updates to Protect Application Availability
Even when configuration changes correctly trigger a rollout, the update must be performed safely.
Kubernetes Deployments support two main strategies:
RollingUpdate is the default strategy.
Consider this configuration:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0
maxSurge: 1
maxUnavailable: 0 prevents the rollout controller from intentionally reducing available replicas below the desired count during the update.
maxSurge: 1 allows one additional Pod to be created above the desired replica count.
For a Deployment with three replicas, Kubernetes can temporarily create a fourth Pod while replacing existing instances.
This helps maintain application capacity during the update, provided the cluster has sufficient resources.
However, configuration compatibility is just as important as Pod availability.
Suppose the old application expects a configuration property named DatabaseConnection, while the new version expects DatabaseConnectionString.
If both versions run simultaneously during a rolling update, they may interpret the configuration differently.
A safer approach is to design configuration changes so that old and new application versions can coexist during the transition.
For example, introduce the new property while temporarily preserving the old one, deploy an application version that supports both, and remove the old property in a later release.
This reduces the risk of breaking existing Pods while new Pods are starting.
Keep Configuration Changes Compatible With Application Versions
A rolling deployment does not replace every Pod at exactly the same moment.
Old and new application versions can run together for a period of time.
This means configuration changes should be designed with backward and forward compatibility in mind.
Consider a payment service URL change.
The old application version expects:
PaymentServiceUrl
The new version expects:
PaymentApiEndpoint
If the configuration is changed before the new application version is deployed, existing Pods may continue working because they already loaded the previous value. However, newly created old-version Pods may fail if they receive only the new property.
A safer migration can follow this sequence:
First, support both configuration properties.
Update the application so it can read the new property and fall back to the old property.
var paymentEndpoint =
builder.Configuration["PaymentApiEndpoint"]
?? builder.Configuration["PaymentServiceUrl"];
if (string.IsNullOrWhiteSpace(paymentEndpoint))
{
throw new InvalidOperationException(
"Payment service endpoint is not configured.");
}
Next, introduce the new configuration property.
Keep the old property available while the application transition is in progress.
Then, deploy the new application version.
Wait until all replicas have been updated and verified.
Finally, remove the old property.
This staged approach prevents configuration changes from becoming tightly coupled to a single application deployment step.
The same principle applies to renamed settings, changed authentication methods, database connection properties, and external service endpoints.
Monitor Configuration Rollouts Before Removing Old Resources
A successful kubectl apply command only confirms that Kubernetes accepted the resource update. It does not prove that every application instance is using the intended configuration successfully.
After a configuration rollout, verify the Deployment:
kubectl get deployment order-api
Check the rollout:
kubectl rollout status deployment/order-api
Inspect the ReplicaSets:
kubectl get replicasets -l app=order-api
Review the Pods:
kubectl get pods -l app=order-api -o wide
Inspect recent events:
kubectl get events --sort-by=.metadata.creationTimestamp
Application-level health checks are also important.
For an ASP.NET Core API, a readiness endpoint can verify whether the application is ready to receive requests.
For example, the application might check whether required configuration values are present during startup and whether essential dependencies are available.
Do not rely only on Kubernetes reporting that a Pod is Running.
A container can be running while using an invalid connection string or failing to communicate with an external service.
Also, avoid deleting old ConfigMaps or Secrets immediately after changing the Deployment. Confirm that no active Pods or rollback revisions still reference them.
Common Mistakes to Avoid
Several configuration practices can make rolling deployments unpredictable.
Assuming a ConfigMap update automatically restarts Pods
Environment variables do not update inside running containers. Trigger a rollout or use a configuration versioning strategy.
Updating a shared ConfigMap without coordinating the Deployment
New Pods may receive different configuration from existing Pods. Make configuration changes and Pod replacement part of a controlled release.
Using subPath mounts when automatic configuration updates are expected
ConfigMap and Secret updates are not propagated to subPath-mounted files.
Assuming mounted files are automatically reloaded by the application
Kubernetes may update projected files, but the application must still detect and reload them.
Changing credentials without an overlap period
Rotating a database password or API key can break existing Pods before replacement Pods become ready.
Deleting old configuration too early
Old ReplicaSets may still need the previous configuration during a rollback or recovery.
Using a checksum that does not represent the actual configuration
If the checksum does not change when the configuration changes, Kubernetes will not detect a new Pod template.
Ignoring application compatibility
A rolling update can run old and new application versions simultaneously. Configuration changes must support that transition.
Summary
Keeping configuration changes safe during a rolling Kubernetes deployment requires understanding how ConfigMaps, Secrets, and Deployment Pod templates interact.
Updating a ConfigMap or Secret does not automatically restart existing containers. Environment variables remain unchanged for the lifetime of the process, while ordinary mounted configuration files can receive updates eventually if the application is designed to reload them.
For applications that read configuration only at startup, a controlled Pod replacement is necessary. A rollout restart, checksum annotation, or versioned ConfigMap and Secret can trigger that replacement.
Production deployments should also account for credential rotation, application compatibility, rolling update settings, and rollback requirements.
The most reliable approach is to treat application configuration as part of the release process rather than as an independent Kubernetes resource update. When configuration changes are versioned, reviewed, and deployed alongside the application, teams can avoid mixed settings across replicas and make production rollouts more predictable.
Join the conversation! Your thoughts help the community grow.