Deploying a new application version to Kubernetes is usually straightforward. You update the container image or configuration, apply the Deployment manifest, and Kubernetes gradually replaces the existing Pods with new ones.
However, sometimes the deployment never completes. The new Pods are created, but they remain in a Pending state, repeatedly restart, fail their readiness checks, or never become available. The Deployment continues waiting, and the application may run with a mixture of old and new versions.
This situation can be frustrating because the Deployment itself may look healthy at first. The configuration was accepted, the ReplicaSet was created, and Kubernetes started scheduling Pods. The actual problem might be inside the container, in its configuration, or in the resources available to the cluster.
Recovering a stuck deployment requires more than repeatedly applying the same YAML file. You need to identify where the rollout is failing, understand why the new Pods are not becoming ready, and decide whether to fix the current version or return to the previous working version.
This article explains how to troubleshoot an unsuccessful Kubernetes rollout, identify common causes, and recover a Deployment without making the situation worse.
Understand What Happens During a Kubernetes Deployment
A Kubernetes Deployment manages application Pods through ReplicaSets. When you update a Deployment's Pod template, Kubernetes creates a new ReplicaSet and gradually moves the application from the old version to the new one.
For example, imagine an application running three replicas:
Three Pods are running the existing application version.
You update the Deployment to use a new container image.
Kubernetes creates a new ReplicaSet.
New Pods are created using the updated image.
Kubernetes waits for the new Pods to become available.
Once the rollout succeeds, the old ReplicaSet is scaled down.
This process is called a rolling update.
The advantage is that Kubernetes can replace application instances gradually instead of terminating every existing Pod at once.
However, a rolling update depends on the new Pods becoming ready. If the application fails to start or its readiness probe keeps failing, Kubernetes cannot complete the rollout.
The old Pods may continue serving traffic, depending on the rollout strategy and available replicas. This is useful because it can prevent an unsuccessful deployment from immediately removing every working instance.
Understanding this behavior is important before attempting recovery. A deployment that has stopped progressing does not necessarily mean that every existing application instance is unavailable.
Identify Whether the Deployment Is Actually Stuck
The first step is to check the Deployment status rather than immediately changing the configuration.
Run the following command:
kubectl get deployments
You might see output similar to this:
NAME READY UP-TO-DATE AVAILABLE AGE
my-api 2/3 1 2 15m
The application expects three replicas, but only two are ready and available. Only one replica has been updated to the new version.
This indicates that the rollout has not completed.
Next, check the rollout status:
kubectl rollout status deployment/my-api
If the deployment is still progressing, you might see a message indicating that Kubernetes is waiting for the updated replicas to become available.
You can also inspect the Deployment:
kubectl describe deployment my-api
Pay particular attention to the Conditions and Events sections.
A stalled rollout may eventually show a condition with the reason ProgressDeadlineExceeded. This means Kubernetes has detected that the Deployment has not made progress within its configured progress deadline.
The default progress deadline is 600 seconds. You can configure a different value through the progressDeadlineSeconds field.
An important distinction is that this deadline does not automatically fix the application or roll back the deployment. It reports that the rollout has failed to make progress within the expected period.
The next step is to identify why the new Pods are not becoming ready.
Inspect the New Pods and Their Status
A Deployment creates ReplicaSets, and ReplicaSets create Pods. Therefore, examining the Pods is usually the most useful next step.
Run:
kubectl get pods -l app=my-api
The label in this command must match the labels used by your Deployment.
You might see output like this:
NAME READY STATUS RESTARTS AGE
my-api-old-7d8f9c6d4a 1/1 Running 0 25m
my-api-old-7d8f9c6d4b 1/1 Running 0 25m
my-api-new-6f7c8d9b5a 0/1 CrashLoopBackOff 5 8m
The old Pods are running, but the new Pod is repeatedly crashing.
The STATUS column provides an initial clue, but it does not explain the complete problem.
Use the following command to inspect the new Pod:
kubectl describe pod my-api-new-6f7c8d9b5a
Review the container state, restart count, conditions, and Events section.
The output may reveal problems such as an image pull failure, insufficient resources, a failed health check, or a configuration issue.
Do not assume that every Pod in a failed rollout has the same problem. If multiple Pods are failing, inspect more than one before deciding on a fix.
Fix ImagePullBackOff and ErrImagePull
One common reason a new Deployment never becomes ready is that Kubernetes cannot download the container image.
The Pod may show one of these statuses:
ErrImagePull
ImagePullBackOff
This usually means the container runtime cannot retrieve the requested image.
Start by checking the image configured in the Deployment:
kubectl describe deployment my-api
Verify the image repository and tag.
For example, a small typing mistake can result in Kubernetes trying to download an image that does not exist.
containers:
- name: my-api
image: myregistry/my-api:1.0.10
If the intended version is 1.0.1, the incorrect tag will prevent the container from starting.
Also check whether the image has actually been pushed to the registry.
If the registry is private, verify that the Deployment has access to the required image pull secret:
spec:
template:
spec:
imagePullSecrets:
- name: registry-secret
The secret must exist in the same namespace as the Pod.
You can verify it with:
kubectl get secrets
If the image name, tag, registry access, or credentials are incorrect, correct the configuration and apply the updated Deployment.
Do not repeatedly delete Pods to solve an image pull problem. Kubernetes will create replacement Pods using the same incorrect image configuration, so the failure will continue.
Investigate CrashLoopBackOff
A Pod in CrashLoopBackOff means its container repeatedly starts and exits. Kubernetes attempts to restart the container, but repeated failures cause increasing delays between restart attempts.
This status often points to an application startup problem rather than a scheduling problem.
Start by checking the container logs:
kubectl logs my-api-new-6f7c8d9b5a
If the container has restarted, the previous container's logs may be more useful:
kubectl logs my-api-new-6f7c8d9b5a --previous
For an application built with ASP.NET Core, the logs might reveal problems such as:
A missing connection string.
An invalid application configuration.
A database connection failure.
An exception during application startup.
A missing environment variable.
A permission error when accessing a required file.
For example, an application might require a database connection string:
env:
- name: ConnectionStrings__DefaultConnection
valueFrom:
secretKeyRef:
name: api-secrets
key: database-connection
If the referenced Secret does not exist or the expected key is missing, the application may fail during startup.
Check the Secret and verify that the Deployment references the correct name and key.
Also distinguish between an application crash and a container termination caused by resource limits. If the Pod reports an OOMKilled reason, investigate memory usage and configured memory limits rather than focusing only on application exceptions.
The container logs, Pod description, and termination reason should be considered together before changing resource settings or application code.
Correct Readiness Probe Failures
A container can be running without being ready to receive traffic.
Kubernetes uses readiness probes to determine whether an application is prepared to serve requests. If a readiness probe fails, the Pod is not considered ready and is excluded from the ready endpoints of matching Services.
Consider this example:
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
This configuration tells Kubernetes to send HTTP requests to the application's health endpoint.
If the application listens on another port, exposes a different health path, or takes longer to initialize, the probe may fail even though the container process is running.
For an ASP.NET Core application, verify that the health endpoint is registered correctly:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks();
var app = builder.Build();
app.MapHealthChecks("/health");
app.Run();
Also verify that the application listens on the port expected by the probe.
A common mistake is changing the container port without updating the readiness probe. The application starts successfully, but Kubernetes continues checking the wrong port.
Do not simply increase the probe delay or failure threshold without understanding the startup behavior. Probe settings should reflect the application's actual initialization requirements.
If the application needs additional time before it can start responding, a startup probe may be more appropriate. It gives the application time to initialize before Kubernetes begins applying the normal liveness and readiness checks.
Readiness and liveness probes serve different purposes. Readiness determines whether a Pod should receive traffic, while liveness determines whether a container should be restarted. Using an aggressive liveness probe to handle slow startup can cause repeated restarts.
Check Resource Requests and Limits
A deployment can also fail because the cluster does not have enough available resources.
For example, a new Pod might remain in Pending state because no suitable node has enough available CPU or memory to satisfy its resource requests.
Check the Pod events:
kubectl describe pod my-api-new-6f7c8d9b5a
Look for messages such as:
FailedScheduling
Insufficient cpu
Insufficient memory
These messages indicate that the scheduler cannot find a suitable node.
Review the resource configuration in the Deployment:
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "500m"
memory: "512Mi"
Requests represent the resources Kubernetes uses when scheduling the Pod. Limits define the maximum resources available to the container for the specified resource.
If the application requires more memory than its configured limit, it may be terminated with an OOMKilled status.
If CPU limits are too restrictive, the application may experience CPU throttling and take longer to respond to readiness checks.
However, increasing resource limits without measuring actual usage is not a reliable fix. First inspect available node capacity, resource requests, container metrics, and recent application behavior.
If the cluster genuinely lacks capacity, you may need to adjust resource requirements, scale the cluster, or reduce competing workloads.
Roll Back to the Previous Working Version
Sometimes the fastest way to restore application stability is to return to the previous working version.
If the new version is causing failures and the issue cannot be fixed quickly, use the Deployment rollback command:
kubectl rollout undo deployment/my-api
Kubernetes restores the previous Deployment revision's Pod template.
You can monitor the rollback:
kubectl rollout status deployment/my-api
To inspect available revisions before rolling back, run:
kubectl rollout history deployment/my-api
You can also inspect a specific revision:
kubectl rollout history deployment/my-api --revision=2
If you need to return to a particular revision, specify it explicitly:
kubectl rollout undo deployment/my-api --to-revision=2
After the rollback, verify the Pods and Deployment status:
kubectl get deployments
kubectl get pods
A rollback is useful for restoring the previous application version, but it does not reverse every change associated with a deployment.
For example, if the new application version changed the database schema, rolling back the container image does not automatically reverse those database changes.
This is why database migrations should be designed with deployment compatibility in mind. Prefer changes that allow the old and new application versions to work with the database during the transition.
Also remember that a rollback only works when the required revision history is available. If deployment history has been removed or limited, the expected revision may not be available.
Understand Rolling Update Settings Before Changing Them
Kubernetes provides two important rolling update settings: maxUnavailable and maxSurge.
maxUnavailable controls how many Pods can be unavailable during an update.
maxSurge controls how many additional Pods can be created above the desired replica count.
For example:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 1
maxSurge: 1
With three desired replicas, this configuration allows Kubernetes to make up to one existing replica unavailable and create up to one additional Pod during the rollout.
These settings influence how the Deployment progresses when new Pods fail to become ready.
If maxUnavailable is too restrictive for the available cluster capacity, the rollout may struggle to progress. If it allows too many Pods to become unavailable, application availability may be affected.
For critical applications, review these values together with the number of replicas, resource availability, readiness probes, and expected traffic.
Changing rolling update settings will not fix a broken container image or an application that cannot start. It only changes how Kubernetes manages the replacement process.
Verify the Application After Recovery
A successful rollout does not automatically prove that the application is functioning correctly.
After fixing the problem or rolling back, verify the Deployment:
kubectl rollout status deployment/my-api
Check the available replicas:
kubectl get deployment my-api
Inspect the Pods:
kubectl get pods -l app=my-api
Also verify the Service endpoints:
kubectl get endpointslice
EndpointSlices show the backend endpoints associated with Services. Check the relevant EndpointSlice and confirm that the expected ready Pods are available.
Finally, test the application through its actual Service or ingress endpoint.
For an API, verify that important operations work, not just that the health endpoint returns a successful response.
Check application logs, error rates, and resource usage after recovery. A deployment may become ready but still experience runtime errors that were not detected by its readiness probe.
If the application depends on a database, message broker, or external service, verify those integrations as well.
Common Mistakes During Kubernetes Deployment Recovery
Mistake | Why it creates problems | Better approach |
|---|
Repeatedly applying the same manifest | Does not fix an incorrect configuration | Identify the failure before applying changes |
Deleting failing Pods without fixing the cause | The ReplicaSet creates replacements with the same problem | Inspect events and container logs |
Increasing readiness probe delays without investigation | Can hide application startup or configuration problems | Verify the health endpoint and startup behavior |
Increasing memory limits without measuring usage | May waste cluster resources without solving the issue | Inspect memory usage and termination reasons |
Rolling back without checking database compatibility | The old application may not work with new schema changes | Use backward-compatible database migrations |
Changing maxUnavailable and maxSurge blindly | Can reduce application availability or prevent progress | Review replica count and cluster capacity |
Checking only whether the container is Running | Running does not mean the application is ready | Verify readiness and actual application behavior |
Best Practices for Reliable Kubernetes Deployments
Recovering a failed deployment is important, but preventing repeated failures is even better.
The following practices help make application releases more predictable:
Use meaningful and traceable container image tags instead of relying on mutable tags such as latest.
Validate image availability and registry credentials before deploying.
Configure readiness, liveness, and startup probes according to actual application behavior.
Set resource requests and limits based on measured application requirements.
Use a suitable rolling update strategy for the application's availability requirements.
Configure progressDeadlineSeconds to detect stalled rollouts within an appropriate period.
Keep deployment history available so that previous versions can be restored when necessary.
Make database migrations compatible with both the existing and new application versions whenever possible.
Include rollout status checks in CI/CD pipelines and stop promotion when the deployment does not become ready.
Monitor application errors, Pod restarts, resource usage, and Service availability after every release.
A deployment pipeline should not consider an update successful merely because Kubernetes accepted the manifest. It should verify that the expected replicas become available and that the application passes its essential health checks.
Summary
A Kubernetes Deployment can remain stuck when new Pods fail to start, cannot download their container images, run out of resources, or repeatedly fail readiness checks.
The most effective recovery approach is to investigate the rollout systematically. Start with the Deployment status, inspect the new Pods, review their events, and examine container logs. Once the actual cause is understood, correct the configuration or application problem and monitor the rollout again.
If the new version cannot be recovered quickly and the application is affected, rolling back to a previous working revision can restore service. However, rollback decisions should also consider database migrations and other changes made during the release.
Reliable Kubernetes deployments depend on more than successful container creation. They require correct health checks, appropriate resource settings, controlled rolling updates, and verification that the application is genuinely ready to serve traffic.
Join the conversation! Your thoughts help the community grow.