Kubernetes 1.34: Testing .NET Workloads Before Maintenance Mode

Introduction

Kubernetes upgrades are rarely difficult because of the version number itself.

The difficult part is everything running on top of Kubernetes.

For .NET teams, that can include ASP.NET Core APIs, background workers, gRPC services, scheduled jobs, ingress controllers, service meshes, autoscaling, persistent storage, observability agents, and deployment automation.

Kubernetes 1.34 introduced several changes that are worth testing against .NET workloads before moving production clusters. Kubernetes follows a release lifecycle in which each minor release receives approximately 12 months of support, making upgrade planning an ongoing operational requirement.

The right strategy is not:

New Kubernetes Version
        |
        v
Upgrade Production

It is:

Current Cluster
      |
      v
Compatibility Tests
      |
      v
Application Tests
      |
      v
Infrastructure Tests
      |
      v
Staging Upgrade
      |
      v
Production Canary
      |
      v
Full Rollout

This article explains how to test .NET workloads against Kubernetes 1.34 before making the upgrade a production dependency.

Why Kubernetes Version Testing Matters

A .NET application does not run in isolation.

A typical production deployment may look like this:

                    Kubernetes Cluster
                           |
        +------------------+------------------+
        |                  |                  |
        v                  v                  v
   ASP.NET Core        Worker Service       CronJob
        |                  |                  |
        +------------------+------------------+
                           |
                    Service / Ingress
                           |
                    Config / Secrets
                           |
                    Storage / Network

The application may continue compiling perfectly while the Kubernetes environment introduces an operational regression.

Examples include:

Changed API behavior
Deprecated resources
Container runtime differences
Scheduling behavior
Resource management changes
Network policy issues
Admission-controller incompatibilities
Autoscaling differences
Monitoring-agent failures

That is why Kubernetes upgrades require infrastructure-level regression testing.

Understand the Kubernetes Release Lifecycle

Kubernetes minor releases have a relatively short support window compared with many application frameworks.

The Kubernetes project documents a release lifecycle of approximately one year for each minor release.

That means cluster upgrades should be treated as a recurring engineering process rather than an occasional emergency.

A healthy upgrade cycle looks like:

Current Version
      |
      v
Compatibility Review
      |
      v
Next Supported Version
      |
      v
Staging Validation
      |
      v
Production Upgrade
      |
      v
Next Upgrade Planning

Waiting until a cluster is close to the end of support creates unnecessary migration pressure.

Inventory the .NET Workloads First

Before testing Kubernetes 1.34, build an inventory.

For example:

Workload	Type	Criticality	Dependencies
Orders API	ASP.NET Core	High	PostgreSQL, Redis
Worker	.NET Worker	High	Queue
Reporting API	ASP.NET Core	Medium	Database
Scheduler	CronJob	Medium	Storage
gRPC API	ASP.NET Core	High	Internal services

Also record:

.NET version
Container base image
CPU requests
Memory requests
CPU limits
Memory limits
Replicas
Probes
Ingress
Service
ConfigMaps
Secrets
PersistentVolumes
HPA
PDB
NetworkPolicies

This inventory becomes your upgrade test matrix.

Verify the Container Images

Kubernetes upgrades do not automatically upgrade your .NET runtime.

Your container image remains responsible for the runtime environment.

For example:

FROM mcr.microsoft.com/dotnet/aspnet:8.0

WORKDIR /app

COPY publish/ .

ENTRYPOINT ["dotnet", "Orders.Api.dll"]

The important compatibility questions are:

Kubernetes 1.34
       |
       v
Container Runtime
       |
       v
Linux Kernel
       |
       v
.NET Runtime
       |
       v
ASP.NET Core

Test the complete chain.

Do not assume that because the .NET application works on the current cluster, it will automatically behave identically after the infrastructure upgrade.

Test Readiness and Liveness Probes

Health probes are one of the first things to validate.

A typical ASP.NET Core deployment might use:

livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  initialDelaySeconds: 5
  periodSeconds: 5

Test:

Application starts
       |
       v
Readiness becomes healthy
       |
       v
Traffic begins
       |
       v
Application failure
       |
       v
Liveness reacts

The upgrade should not change the expected health lifecycle.

Test Graceful Shutdown

.NET applications need to shut down cleanly when Kubernetes terminates a pod.

Test:

Pod receives termination
        |
        v
Application receives shutdown
        |
        v
Stop accepting work
        |
        v
Finish active requests
        |
        v
Release resources
        |
        v
Process exits

For APIs, monitor:

In-flight requests
HTTP 5xx responses
Request completion
Connection cleanup
Background task termination

For workers:

Message acknowledgement
Job completion
Duplicate processing
Queue visibility timeout

A Kubernetes upgrade is a good opportunity to verify graceful termination instead of assuming it works.

Test Resource Requests and Limits

A common .NET deployment configuration is:

resources:
  requests:
    cpu: "250m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "1Gi"

These values influence scheduling and runtime behavior.

Measure before and after the upgrade:

CPU utilization
Memory utilization
CPU throttling
OOMKills
Pod restarts
Request latency
Throughput

Do not simply compare whether the application is "up."

A workload can remain healthy while performance has degraded.

Benchmark ASP.NET Core APIs

Create a baseline before upgrading.

For example:

Endpoint              RPS       p95       p99
------------------------------------------------
GET /orders           1,200     42 ms     85 ms
GET /orders/{id}      1,500     31 ms     64 ms
POST /orders          850       73 ms     140 ms

After moving to Kubernetes 1.34, run the same workload.

The goal is not to prove that every number is identical.

The goal is to identify statistically meaningful regressions.

Track:

Throughput
p50 latency
p95 latency
p99 latency
Error rate
CPU
Memory
Pod restarts

Test Horizontal Pod Autoscaling

Autoscaling is especially important for .NET APIs.

A typical HPA might look like:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: orders-api
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: orders-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Test:

Low Load
   |
   v
2 Pods

High Load
   |
   v
Scale Out

Load Falls
   |
   v
Scale In

Measure:

Scale-up time
Maximum replicas
Scale-down behavior
Request latency during scaling
CPU utilization
Pod startup time

Test Startup Time

Autoscaling only works well if new pods become ready quickly.

Measure:

Image Pull
   +
Container Start
   +
.NET Runtime Start
   +
Application Initialization
   +
Readiness

For a .NET service:

Pod Created
   |
   v
Container Started
   |
   v
.NET Starts
   |
   v
Dependency Initialization
   |
   v
Health Endpoint Ready
   |
   v
Traffic

Compare startup times before and after the cluster upgrade.

Test Deployment Rollouts

Use a controlled deployment:

kubectl rollout status deployment/orders-api

Then test:

Old Pods
   |
   v
New Pods
   |
   v
Readiness
   |
   v
Traffic Shift
   |
   v
Old Pods Terminated

Monitor:

Available replicas
Unavailable replicas
Restart count
HTTP errors
Latency

The upgrade should not introduce unexpected rollout behavior.

Test Pod Disruption Budgets

A production API may use:

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: orders-api
spec:
  minAvailable: 2
  selector:
    matchLabels:
      app: orders-api

Test the behavior during node maintenance and cluster operations.

The objective is:

Node Maintenance
       |
       v
Pod Eviction
       |
       v
Enough replicas remain available

A Kubernetes upgrade should not accidentally turn maintenance into an application outage.

Test Stateful .NET Services Carefully

Stateless APIs are generally easier to upgrade than stateful applications.

If your .NET workload depends on:

PostgreSQL
Redis
RabbitMQ
Kafka
PersistentVolume

test the dependency path separately.

For example:

ASP.NET Core
      |
      v
Database Service
      |
      v
Persistent Storage

Verify:

Connection establishment
Connection pooling
Failover
DNS
Timeouts
Reconnection
Persistent-volume attachment

Test Kubernetes API Compatibility

One of the most important upgrade checks is API compatibility.

Search manifests for deprecated API versions:

grep -R "apiVersion:" ./k8s

Then review every resource used by the application.

Pay particular attention to:

Deployment
Ingress
CronJob
HorizontalPodAutoscaler
PodDisruptionBudget
NetworkPolicy
RBAC

The application code may be perfectly compatible while the Kubernetes manifests are not.

Test Ingress Behavior

For an ASP.NET Core API exposed through an ingress controller, test:

Client
  |
  v
Ingress
  |
  v
Service
  |
  v
ASP.NET Core Pod

Measure:

TLS termination
HTTP routing
Path rewriting
WebSockets
gRPC
Request body size
Timeouts
Client IP forwarding

For gRPC services, explicitly test HTTP/2 behavior.

Do not assume that because ordinary HTTP endpoints work, gRPC is also working correctly.

Test Network Policies

If NetworkPolicy is used, validate the application dependency graph.

For example:

orders-api
    |
    +----> PostgreSQL
    |
    +----> Redis
    |
    +----> payment-service

Test both:

Allowed connections

and:

Blocked connections

Security controls should remain effective after the upgrade.

Test Secrets and Configuration

.NET applications often depend on environment variables:

env:
  - name: ConnectionStrings__Default
    valueFrom:
      secretKeyRef:
        name: orders-secrets
        key: database

Verify:

Secret exists
       |
       v
Pod receives secret
       |
       v
.NET configuration loads
       |
       v
Application connects

Also verify that secrets are not accidentally exposed through logs or diagnostic endpoints.

Test Background Workers

ASP.NET Core is not the only .NET workload that needs testing.

A worker service may look like:

public sealed class OrderWorker : BackgroundService
{
    protected override async Task ExecuteAsync(
        CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            await ProcessOrdersAsync(stoppingToken);
        }
    }
}

Test what happens when Kubernetes terminates the pod.

You want:

CancellationToken
       |
       v
Stop accepting new work
       |
       v
Finish current operation
       |
       v
Commit/Acknowledge
       |
       v
Exit

Otherwise, a rolling upgrade can cause duplicate jobs or lost work.

Test CronJobs

For scheduled .NET workloads, verify:

Schedule
Concurrency Policy
Job Completion
Failure Retry
History Limits
Pod Cleanup

A simple CronJob might run:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: daily-report
spec:
  schedule: "0 2 * * *"
  concurrencyPolicy: Forbid

Make sure the job still behaves correctly during node maintenance and upgrades.

Test Observability

A cluster upgrade should include an observability regression test.

Monitor:

Logs
Metrics
Traces
Events
Pod status
Node status
Application health

For .NET applications using OpenTelemetry, verify:

ASP.NET Core request
      |
      v
Trace
      |
      +----> Database span
      |
      +----> HTTP span
      |
      +----> Queue span

Do not treat observability as optional.

If telemetry disappears after an upgrade, diagnosing production issues becomes much harder.

Test Node-Level Behavior

The application runs on nodes, so test node characteristics as well.

Inspect:

kubectl get nodes
kubectl describe node <node-name>

Review:

Kubernetes version
Container runtime
CPU capacity
Memory capacity
Conditions
Taints
Labels

Your workloads may depend on node labels or affinity rules.

For example:

affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
        - matchExpressions:
            - key: workload
              operator: In
              values:
                - application

Verify that the scheduling constraints still produce the expected placement.

Build a Staging Cluster

The most valuable test is a production-like staging environment.

Use:

Same .NET version
Same container images
Same Helm charts
Same manifests
Same ingress
Same policies
Same observability
Same autoscaling

Only the Kubernetes version should intentionally differ.

Architecture:

Production
Kubernetes Current
       |
       | Same Workload
       |
       v
Staging
Kubernetes 1.34

This creates a meaningful comparison.

Use a Test Matrix

A useful upgrade matrix is:

Area	Test	Pass Criteria
Build	Container starts	No startup failures
Health	Readiness	Healthy
API	Load test	Within threshold
Database	CRUD	No errors
Worker	Job processing	No lost/duplicate jobs
HPA	Scale test	Expected scaling
Ingress	HTTP/gRPC	Successful
Security	NetworkPolicy	Expected access
Storage	Volume	Read/write works
Observability	Logs/traces	Available
Deployment	Rollout	No outage

This turns "Kubernetes upgrade testing" into measurable engineering work.

Run Failure Tests

Do not test only the happy path.

Test:

Pod crash
Node failure
Database unavailable
Network timeout
High CPU
High memory
Slow startup
Failed readiness
Failed liveness
Rolling restart

For example:

Database unavailable
        |
        v
ASP.NET Core
        |
        v
Connection Failure
        |
        v
Retry / Failure Handling
        |
        v
Recovery

The upgraded cluster should preserve the application's existing resilience behavior.

Canary the Production Upgrade

After staging passes, do not immediately upgrade every node.

Use a controlled rollout.

Production Cluster
       |
       v
Upgrade Small Node Group
       |
       v
Move Selected Workloads
       |
       v
Observe
       |
       v
Expand

Monitor:

Error rate
Latency
CPU
Memory
Pod restarts
Scheduling failures
Application logs
Kubernetes events

Only continue if the signals remain healthy.

Automate Upgrade Regression Tests

Manual testing does not scale.

A useful pipeline can be:

Infrastructure Change
        |
        v
Create Test Cluster
        |
        v
Deploy .NET Workloads
        |
        v
Run Integration Tests
        |
        v
Run Load Tests
        |
        v
Run Failure Tests
        |
        v
Collect Metrics
        |
        v
Pass / Fail

For application-level tests, standard .NET tooling can remain unchanged.

For example:

dotnet test

can validate application behavior while Kubernetes-specific checks validate deployment behavior.

Common Mistakes

Testing Only `kubectl get pods`

A green pod does not mean a healthy application.

Testing Only Startup

A workload can start correctly and fail under load.

Ignoring Autoscaling

Scaling behavior can be just as important as application correctness.

Ignoring Graceful Shutdown

Rolling upgrades exercise termination paths heavily.

Testing Only HTTP

gRPC, WebSockets, queues, and background workers need separate validation.

Skipping Observability

You need telemetry to understand upgrade behavior.

Upgrading Production First

Production should be the final validation environment, not the first.

Best Practices

Use a repeatable checklist:

[ ] Inventory .NET workloads
[ ] Inventory Kubernetes resources
[ ] Review deprecated APIs
[ ] Verify container images
[ ] Test readiness/liveness
[ ] Test graceful shutdown
[ ] Test resource limits
[ ] Benchmark APIs
[ ] Test HPA
[ ] Test deployments
[ ] Test PDB
[ ] Test ingress
[ ] Test gRPC
[ ] Test NetworkPolicy
[ ] Test secrets
[ ] Test background workers
[ ] Test CronJobs
[ ] Test observability
[ ] Test node scheduling
[ ] Run staging upgrade
[ ] Run failure tests
[ ] Run production canary
[ ] Monitor before full rollout

Advantages and Disadvantages

Advantages

Detects infrastructure regressions before production.
Protects .NET application availability.
Validates deployment manifests and APIs.
Provides measurable upgrade confidence.
Reduces emergency maintenance work.
Creates a repeatable Kubernetes upgrade process.

Disadvantages

Requires a representative staging environment.
Load testing infrastructure takes time to maintain.
Some infrastructure behavior is difficult to reproduce exactly.
Production canaries require careful operational planning.
Kubernetes upgrades can expose problems in third-party controllers and add-ons outside the application team's direct control.

Final Thoughts

Kubernetes upgrades should be treated as compatibility projects, not simply version changes.

For .NET teams, the application code is only one part of the upgrade surface. ASP.NET Core startup behavior, health probes, graceful shutdown, resource allocation, autoscaling, ingress, gRPC, background workers, storage, network policies, and observability all need to be validated.

Kubernetes' roughly one-year support lifecycle makes this especially important because waiting until maintenance pressure becomes urgent leaves less time for testing.

The safest approach is straightforward: establish a baseline on the current cluster, create a production-like Kubernetes 1.34 environment, run the same .NET workloads, measure application and infrastructure behavior, deliberately test failure scenarios, and then move through a controlled production canary.

The goal is not simply to prove that your .NET application runs on Kubernetes 1.34.

The goal is to prove that it still behaves correctly under production conditions after the upgrade.

Introduction

Why Kubernetes Version Testing Matters

Understand the Kubernetes Release Lifecycle

Inventory the .NET Workloads First

Verify the Container Images

Test Readiness and Liveness Probes

Test Graceful Shutdown

Test Resource Requests and Limits

Benchmark ASP.NET Core APIs

Test Horizontal Pod Autoscaling

Test Startup Time

Test Deployment Rollouts

Test Pod Disruption Budgets

Test Stateful .NET Services Carefully

Test Kubernetes API Compatibility

Test Ingress Behavior

Test Network Policies

Test Secrets and Configuration

Test Background Workers

Test CronJobs

Test Observability

Test Node-Level Behavior

Build a Staging Cluster

Use a Test Matrix

Run Failure Tests

Canary the Production Upgrade

Automate Upgrade Regression Tests

Common Mistakes

Testing Only kubectl get pods

Testing Only Startup

Ignoring Autoscaling

Ignoring Graceful Shutdown

Testing Only HTTP

Skipping Observability

Upgrading Production First

Best Practices

Advantages and Disadvantages

Advantages

Disadvantages

Final Thoughts

Testing Only `kubectl get pods`