Kubernetes 1.34: Building a Post-Maintenance Upgrade Validation Suite for .NET

Introduction

Upgrading Kubernetes is not finished when the cluster reports that every node is running the new version.

The more important question comes afterward:

Are the applications, networking, storage, workloads, and deployment processes still working as expected?

This matters even more for .NET applications running in Kubernetes. A cluster upgrade can complete successfully while an application later exposes problems involving readiness probes, service discovery, ingress, configuration, persistent storage, or deployment behavior.

Kubernetes 1.34 provides a useful opportunity to build a repeatable post-upgrade validation process rather than relying on a few manual checks.

The goal is not to prove that the Kubernetes upgrade itself succeeded.

The goal is to verify that the application platform remains healthy after the upgrade.

What Is Post-Maintenance Validation?

Post-maintenance validation is a collection of automated checks performed after infrastructure maintenance.

For a Kubernetes cluster hosting .NET applications, the workflow can look like this:

Kubernetes Upgrade
        |
        v
Cluster Health
        |
        v
Node Health
        |
        v
Workload Health
        |
        v
Networking
        |
        v
Application Tests
        |
        v
Deployment Validation
        |
        v
Production Ready

This is different from checking only:

kubectl get nodes

A healthy node does not guarantee that the application running on it is healthy.

Why .NET Applications Need Application-Level Checks

Consider an ASP.NET Core application deployed to Kubernetes:

Internet
   |
   v
Ingress
   |
   v
Kubernetes Service
   |
   v
ASP.NET Core Pods
   |
   v
Database

After an infrastructure upgrade, every layer can appear healthy independently while the complete request path is broken.

For example:

Pod -> Running
Service -> Exists
Ingress -> Exists

But:

Client -> API = Failure

Application-level validation catches this type of problem.

Build a Validation Suite Before the Upgrade

The best time to create the validation suite is before maintenance begins.

A basic repository structure could be:

k8s-validation/
    cluster/
    workloads/
    networking/
    storage/
    application/
    deployment/

Each directory can contain checks for a different part of the platform.

For example:

cluster/
    nodes.sh

workloads/
    pods.sh
    deployments.sh

networking/
    services.sh
    ingress.sh

application/
    health-check.sh
    smoke-test.sh

The exact structure is not important.

What matters is that the checks are repeatable.

Step 1: Validate Cluster Nodes

Start with:

kubectl get nodes

Then inspect node details:

kubectl get nodes -o wide

A basic validation script can check whether all expected nodes are ready.

For example:

kubectl get nodes --no-headers |
awk '$2 != "Ready" {print "Node not ready:", $1; failed=1}
     END {exit failed}'

The script exits with a non-zero status if a node is not ready.

That makes it suitable for automation.

Step 2: Check System Workloads

Kubernetes system components should also be checked.

For example:

kubectl get pods -n kube-system

Look for:

Pods stuck in Pending
Pods repeatedly restarting
Failed containers
Unexpectedly missing components

A useful command is:

kubectl get pods -A

This provides a broader view of workload state across namespaces.

Do not assume that a Running pod means the application inside it is functioning correctly.

Step 3: Check Deployment Status

For an ASP.NET Core application:

kubectl get deployments -n production

Then inspect the deployment:

kubectl describe deployment my-api -n production

The important values include:

Desired replicas
Available replicas
Updated replicas
Ready replicas

A deployment should have the expected number of available replicas.

Step 4: Check Pods

Inspect the application's pods:

kubectl get pods -n production -l app=my-api

A healthy result might look conceptually like:

NAME                         READY   STATUS
my-api-7f8d9f6d7b-a1b2c      1/1     Running
my-api-7f8d9f6d7b-d3e4f      1/1     Running
my-api-7f8d9f6d7b-g5h6j      1/1     Running

The important part is not simply Running.

Check the READY column and restart counts as well.

For additional information:

kubectl get pods -n production -o wide

Step 5: Check Container Restarts

A pod can be running while a container has restarted repeatedly.

Check:

kubectl get pods -n production

Then inspect a suspicious pod:

kubectl describe pod <pod-name> -n production

Look at the container state and restart count.

A post-upgrade validation suite should flag unexpected increases in restarts.

Step 6: Validate ASP.NET Core Health Checks

Kubernetes infrastructure checks are not enough.

ASP.NET Core applications should expose appropriate health endpoints.

For example:

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddHealthChecks();

var app = builder.Build();

app.MapHealthChecks("/health");

app.Run();

The validation suite can then test:

curl --fail http://api.example.local/health

A successful HTTP response is useful, but the health endpoint should represent meaningful application health rather than merely proving that the process is alive.

Liveness vs Readiness

ASP.NET Core applications commonly use different health concepts.

Liveness

Liveness answers:

Is the application process alive?

A liveness probe can help Kubernetes determine whether a container needs to be restarted.

Readiness

Readiness answers:

Can this application currently receive traffic?

A readiness check is especially important after maintenance.

For example, an application may be running but temporarily unable to connect to a required dependency.

That should not necessarily mean Kubernetes should send it production traffic.

Example Kubernetes Probes

A deployment might use:

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

The actual values should be based on the application's startup and dependency behavior.

Do not copy probe timings blindly into production.

Step 7: Test Service Discovery

Check Kubernetes services:

kubectl get services -n production

Then validate that the expected service exists and exposes the expected ports.

For example:

apiVersion: v1
kind: Service
metadata:
  name: my-api
spec:
  selector:
    app: my-api
  ports:
    - port: 80
      targetPort: 8080

A post-maintenance test should verify that the service still routes traffic to the intended pods.

Step 8: Validate Ingress

If the .NET application is exposed through an ingress controller, test the complete path.

Client
  |
  v
Ingress
  |
  v
Service
  |
  v
Pod
  |
  v
ASP.NET Core

A basic smoke test could be:

curl --fail https://api.example.com/health

Then test an actual API endpoint:

curl --fail https://api.example.com/api/products

The health endpoint proves that the application is reachable.

The API test provides stronger evidence that routing and application behavior are working.

Step 9: Test Database Connectivity

A .NET API may be healthy while database connectivity is broken.

For example:

API
 |
 X
Database

The application may still answer a basic liveness request.

Therefore, the validation suite should include a controlled endpoint or application-level health check that verifies the required database dependency.

For example:

builder.Services.AddHealthChecks()
    .AddSqlServer(
        builder.Configuration.GetConnectionString("Default"));

The exact health-check implementation depends on the database provider.

Do not expose sensitive database details through the public response.

Step 10: Test External Dependencies

The same approach applies to important external services.

Examples include:

Redis
Message brokers
Object storage
Internal APIs
Authentication services

A useful validation flow is:

API
 |
 +--> Database
 |
 +--> Cache
 |
 +--> Message Queue
 |
 +--> External API

Only test dependencies that are genuinely required for the application's production path.

Otherwise, a temporary non-critical service outage could incorrectly mark the entire application as unavailable.

Step 11: Validate Configuration

Kubernetes configuration can come from:

ConfigMaps
Secrets
Environment variables
Mounted files

Check that expected configuration objects exist:

kubectl get configmaps -n production
kubectl get secrets -n production

Do not print secret values as part of the validation output.

The goal is to confirm that required objects exist and that the application starts successfully with them.

Step 12: Test Persistent Storage

Applications using persistent volumes should be tested separately.

Check:

kubectl get pv
kubectl get pvc -n production

Verify that expected claims are bound:

PVC
 |
 v
Bound
 |
 v
Application Pod

Then perform a controlled read/write test if the application requires persistent storage.

A database or production data volume should never be used for destructive validation.

Step 13: Test Scaling

A simple deployment scaling test can verify that the application continues to schedule correctly.

For example:

kubectl scale deployment my-api \
  --replicas=3 \
  -n production

Then:

kubectl rollout status deployment/my-api \
  -n production

Verify that all expected replicas become ready.

After testing, return the deployment to its intended replica count.

The actual test should be performed in an environment where temporary scaling is safe.

Step 14: Test Rolling Updates

A post-upgrade validation suite should verify that the normal deployment process still works.

For example:

kubectl rollout status deployment/my-api \
  -n production

Then inspect:

kubectl rollout history deployment/my-api \
  -n production

The purpose is to verify that application deployment remains functional after the Kubernetes maintenance.

Step 15: Run a Real Smoke Test

Infrastructure validation should finish with an end-to-end test.

For example:

Client
  |
  v
Ingress
  |
  v
Service
  |
  v
ASP.NET Core API
  |
  +--> Database
  |
  +--> Cache

A smoke test could:

Authenticate using a test account.
Request a known API endpoint.
Create a test record if appropriate.
Retrieve it.
Verify the expected response.
Clean up the test data.

The test should avoid modifying real customer data.

Building an Automated Validation Script

A simple shell script can orchestrate checks:

#!/usr/bin/env bash

set -euo pipefail

echo "Checking nodes..."
kubectl get nodes

echo "Checking production pods..."
kubectl get pods -n production

echo "Checking deployments..."
kubectl rollout status \
  deployment/my-api \
  -n production

echo "Checking application health..."
curl --fail https://api.example.com/health

echo "Validation completed successfully."

This is only a starting point.

A mature validation suite should provide clearer failure messages and separate infrastructure, application, and dependency checks.

Creating a Validation Matrix

A matrix makes the post-maintenance result easier to review.

Area	Validation	Expected Result
Nodes	Node readiness	All expected nodes ready
System	System pods	No unexpected failures
Deployment	Replica status	Expected replicas available
Pods	Ready state	All required pods ready
Services	Service routing	Traffic reaches application
Ingress	External request	Successful response
Health	Readiness endpoint	Healthy
Database	Connectivity	Successful
Storage	PVC state	Required claims bound
Deployment	Rollout	Successful
API	Smoke test	Expected response

The actual expected values should come from the application's deployment requirements.

Testing Before and After Maintenance

The strongest approach is to run the same suite before and after maintenance.

Before Upgrade
      |
      v
Validation Suite
      |
      v
Baseline
      |
      v
Kubernetes Maintenance
      |
      v
Same Validation Suite
      |
      v
Compare Results

This gives you a reference point.

If a test was already failing before maintenance, it should not automatically be classified as an upgrade regression.

Capturing Baseline Results

Store the validation output as an artifact.

For example:

validation/
    before/
        cluster.txt
        workloads.txt
        application.txt
    after/
        cluster.txt
        workloads.txt
        application.txt

For application tests, store structured results where possible.

This makes future maintenance comparisons easier.

Common Mistakes

Checking Only Node Status

A ready node does not prove that the application works.

Checking Only Pod Status

A Running pod can still have broken dependencies or application-level failures.

Skipping External Traffic Tests

Internal Kubernetes checks do not validate ingress and external routing.

Ignoring Persistent Storage

Storage problems can remain hidden until the application attempts to read or write data.

Printing Secrets

Validation scripts should never dump secret values into logs.

Testing Only Infrastructure

The application itself must be part of post-maintenance validation.

Running Destructive Tests

Never perform uncontrolled write or delete operations against production data simply to prove that the system works.

Troubleshooting Failed Validation

When a check fails, classify the failure first.

Validation Failure
       |
       +--> Cluster
       |
       +--> Workload
       |
       +--> Network
       |
       +--> Application
       |
       +--> Dependency
       |
       +--> Configuration

For a failed pod, inspect:

kubectl describe pod <pod-name> -n production

Then check logs:

kubectl logs <pod-name> -n production

For deployments:

kubectl describe deployment my-api \
  -n production

For service routing:

kubectl describe service my-api \
  -n production

This provides a systematic troubleshooting path instead of restarting components randomly.

Production Rollout Strategy

A good maintenance process separates upgrade completion from application approval.

Kubernetes Upgrade Complete
          |
          v
Infrastructure Validation
          |
          v
Application Validation
          |
          v
Smoke Tests
          |
          v
Monitoring Review
          |
          v
Production Approval

Do not declare the maintenance successful solely because the Kubernetes control plane and nodes report healthy status.

Best Practices

Build the Suite Before Maintenance

Do not create validation checks while an incident is already happening.

Run the Same Tests Before and After

This creates a meaningful baseline.

Validate the Full Request Path

Test ingress, service, pod, application, and critical dependencies.

Test Application Health

Use meaningful readiness and liveness checks.

Keep Tests Safe

Use synthetic accounts and controlled test data.

Separate Critical and Non-Critical Checks

A failure in a critical database dependency should carry more weight than a failure in an optional development service.

Store Validation Results

Keep machine-readable results where possible so future maintenance can be compared automatically.

Review Monitoring After the Upgrade

Some problems appear only after traffic returns to normal levels.

Advantages

Provides repeatable evidence that applications remain healthy after maintenance.
Reduces dependence on manual verification.
Detects problems beyond basic Kubernetes cluster health.
Can be integrated into deployment and maintenance workflows.
Creates a reusable validation framework for future upgrades.
Helps distinguish infrastructure failures from application failures.

Disadvantages

Requires development effort before the first upgrade.
Tests need ongoing maintenance as applications change.
End-to-end validation can take additional time.
Some failures may require human investigation.
Production smoke tests must be designed carefully to avoid modifying real data.

Conclusion

A Kubernetes upgrade should not be considered successful simply because the cluster returns to a healthy state.

For .NET applications, the real validation starts after the infrastructure maintenance finishes.

Nodes, workloads, services, ingress, health checks, databases, storage, configuration, and deployment workflows should all be tested. The strongest approach is to build these checks before the upgrade, execute them against the existing environment, save the results, and then run the same suite after maintenance.

This gives the team a clear comparison:

Before Maintenance
       |
       v
Known Healthy State
       |
       v
Kubernetes Upgrade
       |
       v
Same Validation Suite
       |
       v
Post-Maintenance State

The goal is not to prove that every internal component is unchanged.

The goal is to prove that the application platform still delivers the behavior the business depends on.

A repeatable post-maintenance validation suite turns Kubernetes upgrades from a manual "looks healthy" exercise into a measurable engineering process.