How to Test Kubernetes Pod Disruptions Before a Cluster Upgrade

Kubernetes cluster upgrades can introduce changes to nodes, networking, storage, and workload scheduling. Even when an upgrade completes successfully, applications may experience temporary downtime if pods cannot restart, replacement pods remain pending, or traffic is sent to unhealthy instances.

One way to reduce these risks is to test how applications respond to pod disruptions before upgrading the cluster.

A pod disruption can happen when a node is drained for maintenance, a workload is rescheduled, a pod is evicted, or an application instance becomes unavailable. Some disruptions are voluntary, such as those caused by a planned node drain. Others are involuntary, such as a node failure or an unexpected process crash.

Kubernetes provides several mechanisms to manage planned disruptions, including PodDisruptionBudgets, readiness probes, replica management, and graceful termination. However, these features work together only when they are configured correctly.

In this article, we will explore how to test pod disruptions, validate PodDisruptionBudgets, simulate node maintenance, check application availability, and identify problems before a production cluster upgrade.

Why Test Pod Disruptions Before an Upgrade?

A successful cluster upgrade does not necessarily mean that every application will remain available throughout the process.

Consider a deployment with three replicas. Under normal conditions, all three pods handle incoming requests. During a node upgrade, Kubernetes may need to evict a pod and start a replacement elsewhere.

If the remaining replicas have enough capacity and continue passing their readiness probes, users may not notice the disruption.

However, several problems can interfere with this process:

A deployment has only one replica.
A PodDisruptionBudget prevents the required eviction.
Replacement pods cannot be scheduled because the cluster lacks capacity.
A readiness probe marks an application ready before it can handle real traffic.
A termination handler does not allow enough time for in-flight requests to finish.
A storage volume cannot be attached to the replacement pod quickly.
An application depends on local state that disappears when the pod terminates.

Testing these conditions before a cluster upgrade helps identify availability risks while there is still time to correct them.

Understand Voluntary and Involuntary Disruptions

Not all pod disruptions behave the same way.

Voluntary Disruptions

Voluntary disruptions are initiated by an intentional action or a Kubernetes operation that can respect disruption controls.

Examples include:

Draining a node for maintenance.
Evicting a pod through the Kubernetes Eviction API.
Performing a node upgrade that requires workloads to move elsewhere.

A PodDisruptionBudget can limit certain voluntary evictions made through the Eviction API. This helps protect a workload from losing too many available replicas during planned maintenance.

However, a PodDisruptionBudget does not prevent every type of disruption. For example, directly deleting a pod or a node failure may bypass the normal voluntary eviction protections.

Involuntary Disruptions

Involuntary disruptions happen without a planned eviction, such as when a node crashes, a machine loses power, or a process terminates unexpectedly.

A PodDisruptionBudget does not prevent these events. It can still help preserve availability during subsequent voluntary operations, but it cannot guarantee that a minimum number of replicas will remain available after an unexpected failure.

This distinction matters when designing a test plan. Planned maintenance tests validate eviction behavior, while failure simulations validate how the application recovers when a disruption occurs outside those protections.

Inspect the Application Before Testing

Before simulating disruptions, inspect the workload's current configuration.

Start by listing deployments in the target namespace:

kubectl get deployments -n production

Check the pods:

kubectl get pods -n production -o wide

The wide output includes node placement, which helps identify whether multiple replicas are running on the same node.

Inspect a specific deployment:

kubectl describe deployment myapplication -n production

Review the replica count, pod template, readiness probes, resource requests, and recent events.

You should also check the available nodes:

kubectl get nodes

If all replicas are concentrated on one node, draining that node may make the entire application unavailable. Multiple replicas do not automatically provide high availability when they share the same failure domain.

For important workloads, consider pod anti-affinity or topology spread constraints to distribute replicas across suitable nodes or availability zones.

Configure a PodDisruptionBudget

A PodDisruptionBudget, commonly called a PDB, specifies how many replicas of a selected workload must remain available during supported voluntary disruptions.

Suppose an application has three replicas and should retain at least two available replicas during planned maintenance.

A suitable PDB could look like this:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: myapplication-pdb
  namespace: production
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: myapplication

The selector must match the labels on the application pods. If it does not, the PDB will not protect the intended workload.

Apply the configuration:

kubectl apply -f myapplication-pdb.yaml

Inspect the result:

kubectl get pdb -n production

For more detail:

kubectl describe pdb myapplication-pdb -n production

The output includes information such as the expected pod count, current healthy pods, desired healthy pods, and allowed disruptions.

For a deployment with three replicas and a minimum availability requirement of two, one healthy pod may generally be disrupted at a time, assuming all three replicas are healthy and the selector matches the intended pods.

If the application has only two replicas and both are healthy, the same PDB normally allows one pod to be disrupted while keeping at least one healthy replica. Whether that is sufficient depends on the application's actual availability and capacity requirements.

Choose Between minAvailable and maxUnavailable

Kubernetes supports two common ways to define a PodDisruptionBudget.

Setting	Meaning	Suitable use
minAvailable	Specifies the minimum number or percentage of selected pods that must remain available.	Applications with a minimum availability requirement.
maxUnavailable	Specifies the maximum number or percentage of selected pods that may be unavailable due to supported voluntary disruptions.	Applications where a bounded number of unavailable replicas is acceptable.

For example:

spec:
  maxUnavailable: 1

This permits at most one selected pod to be unavailable due to disruptions managed by the PDB, subject to the current workload state.

Use either minAvailable or maxUnavailable in a single PDB, not both.

A PDB is not a substitute for replica management. It does not create replacement pods or guarantee that a replacement can be scheduled. The workload controller, available cluster capacity, and application health determine whether replacements become available.

Test Pod Disruptions with a Controlled Node Drain

Draining a node is one of the most practical ways to test how a workload behaves during planned maintenance.

Perform this test in a staging environment first. Draining a node in production can disrupt applications and other workloads scheduled on that node.

Identify a suitable node:

kubectl get nodes

Inspect which pods are running on it:

kubectl get pods -A -o wide

Before draining, verify that the selected node does not host critical workloads that are outside the test scope.

Then drain the node:

kubectl drain worker-node-1 \
  --ignore-daemonsets

This command attempts to evict eligible pods from the node. It respects applicable eviction and disruption controls, including relevant PodDisruptionBudgets.

If the command cannot proceed because a PDB would be violated, do not immediately force the drain. Investigate the workload's available replicas and disruption requirements.

A drain may also be blocked by pods that are not managed by a suitable controller or by other cluster configuration issues. The exact behavior depends on the workload and cluster configuration.

Once the drain completes, inspect the application:

kubectl get pods -n production -o wide

Check whether replacement pods are running and whether they become ready.

When the maintenance simulation is complete, return the node to service:

kubectl uncordon worker-node-1

Uncordoning makes the node schedulable again. It does not guarantee that existing workloads will automatically move back to it.

Measure Application Availability During the Disruption

Pod status alone does not tell you whether users experienced an outage.

A pod can be Running but not Ready. Likewise, a deployment can show available replicas while requests still fail because of an application-level issue.

Monitor both Kubernetes health and real application behavior.

Check Readiness and Deployment Status

Watch the pods during the test:

kubectl get pods -n production --watch

Inspect deployment availability:

kubectl rollout status deployment/myapplication \
  -n production \
  --timeout=180s

The rollout status command reports whether the deployment has completed its current rollout. It is not a complete disruption test and does not prove that all user requests succeeded.

For a more useful test, monitor the application's health endpoint and send representative requests throughout the drain.

For example, a shell loop can repeatedly call an application health endpoint:

while true; do
  date -Is
  curl -sS -o /dev/null \
    -w 'HTTP %{http_code} total=%{time_total}s\n' \
    --max-time 5 \
    https://staging.example.com/health
  sleep 1
done

Replace the example hostname with the staging application's endpoint.

This test records the response status and duration. It can help identify request failures or latency increases during pod replacement.

A successful health response is not necessarily proof that the entire application is working correctly. If possible, also test a representative business operation and monitor error rates, latency, and in-flight requests.

Test Graceful Pod Termination

When Kubernetes terminates a pod, it normally gives the container a grace period to shut down before forcibly terminating it.

Applications that handle HTTP requests, background jobs, or message processing should use this time to stop accepting new work and complete or safely release work already in progress.

A deployment might define:

spec:
  terminationGracePeriodSeconds: 30
  containers:
    - name: web
      image: myapplication:latest

The grace period gives the process time to shut down, but it does not automatically make the application graceful.

The application must respond appropriately to termination signals, and its shutdown time must fit within the configured grace period.

Test the following behaviors:

New requests stop reaching a terminating instance when it is no longer ready.
In-flight requests can complete within the shutdown window.
Background jobs are completed or safely returned to a queue.
Connections are closed cleanly.
The application does not depend on temporary local state that disappears after termination.

Also review readiness probes and application shutdown behavior together. Marking a process unready and stopping it from accepting new work should happen early enough to reduce failed requests during termination.

Check Scheduling Capacity and Pod Placement

A disruption test can reveal that replacement pods cannot be scheduled even when the PDB allows eviction.

For example, a cluster may have enough total CPU capacity but lack sufficient capacity on the nodes eligible to run a particular workload.

Common causes include:

CPU or memory requests that are too large for available nodes.
Node selectors or affinity rules that restrict placement.
Taints and tolerations that prevent scheduling.
Persistent volumes that constrain pod placement.
Resource quotas or namespace limits.
Topology spread requirements that cannot be satisfied.

If a replacement pod remains Pending, inspect it:

kubectl describe pod myapplication-pod \
  -n production

Review the Events section for scheduling errors.

A PDB can control whether a supported voluntary eviction is allowed, but it does not guarantee that a replacement pod can run. Confirm that the cluster has sufficient capacity to handle the expected maintenance scenario.

Common Problems and How to Diagnose Them

Symptom	Possible cause	What to check
Node drain is blocked	A PDB does not permit another disruption.	Healthy replicas, PDB status, and workload availability.
Replacement pod remains Pending	Insufficient resources or scheduling constraints.	Pod events, resource requests, affinity, taints, and volume requirements.
Requests fail during termination	The application does not shut down gracefully.	Readiness probes, termination handling, and grace period.
All replicas become unavailable	Replicas share a node or failure domain.	Pod placement, anti-affinity, and topology spread constraints.
Pod is Running but requests fail	Application health differs from container state.	Readiness probes, logs, service endpoints, and application metrics.
A PDB appears ineffective	The selector does not match the intended pods or the disruption bypasses normal eviction handling.	Pod labels, PDB selector, and the operation used to terminate the pod.

Common Mistakes to Avoid

Testing Only Whether Pods Restart

A restarted pod does not prove that the application remained available. Measure real request success rates and latency throughout the disruption.

Assuming a PDB Prevents Every Outage

A PDB helps control supported voluntary evictions. It does not prevent node failures, guarantee sufficient capacity, or stop every possible method of deleting a pod.

Forcing a Drain When It Is Blocked

A blocked drain may reveal a genuine availability problem. Investigate the cause before overriding protections or deleting workloads manually.

Ignoring Storage and Stateful Workloads

Applications that use persistent volumes may have additional scheduling and attachment constraints. Stateful workloads should be tested according to their storage and recovery requirements.

Testing Only in a Healthy Cluster

A drain test with all replicas healthy may not reveal what happens when one replica is already failing. In staging, test carefully controlled scenarios where the workload begins with reduced capacity, provided the test cannot affect unrelated applications.

Best Practices Before a Cluster Upgrade

Use a repeatable checklist before upgrading production nodes:

Confirm that critical deployments have enough replicas.
Review PDB selectors and allowed disruptions.
Verify that replicas are distributed across suitable failure domains.
Confirm that replacement pods can be scheduled with the remaining capacity.
Test readiness probes and graceful termination.
Monitor real requests during a controlled drain.
Inspect scheduling events and application logs.
Validate storage behavior for stateful workloads.
Check cluster-specific upgrade guidance and any version compatibility requirements.
Define clear abort conditions if error rates or latency exceed acceptable limits.
Keep a recovery plan for workloads that do not return to a healthy state.

Record the test results, including the number of failed requests, peak latency, replacement pod startup time, and any blocked evictions. These measurements provide a baseline for future upgrades.

Advantages and Limitations of Pod Disruption Testing

Aspect	Advantage	Limitation
Availability validation	Reveals whether applications remain accessible during planned maintenance.	A staging test may not reproduce production traffic or infrastructure conditions exactly.
PDB validation	Confirms whether supported voluntary evictions respect availability requirements.	Does not prevent involuntary disruptions or guarantee replacement capacity.
Scheduling validation	Exposes resource constraints and placement problems.	Results depend on the nodes and workload mix present during testing.
Graceful shutdown testing	Identifies request failures and interrupted background work.	Requires realistic application traffic and representative processing workloads.
Upgrade readiness	Provides evidence for maintenance planning.	Cannot guarantee that every failure scenario has been tested.

Summary

Testing Kubernetes pod disruptions before a cluster upgrade helps identify availability problems while there is still time to fix them.

Start by inspecting deployments, replica placement, readiness probes, and PodDisruptionBudgets. In a controlled environment, drain a node and observe whether replacement pods become ready, requests continue succeeding, and applications shut down gracefully.

Do not rely only on pod status. Monitor actual application requests, latency, error rates, scheduling events, and storage behavior. A PDB can limit supported voluntary disruptions, but it cannot guarantee that replacement pods will run or prevent every type of failure.

The objective is to confirm that workloads can tolerate the disruptions expected during maintenance. With repeatable tests, sufficient scheduling capacity, appropriate disruption policies, and clear recovery procedures, cluster upgrades become easier to plan and less likely to cause unexpected application downtime.