Introduction
Kubernetes upgrades are rarely difficult because of the version number itself.
The difficult part is everything running on top of Kubernetes.
For .NET teams, that can include ASP.NET Core APIs, background workers, gRPC services, scheduled jobs, ingress controllers, service meshes, autoscaling, persistent storage, observability agents, and deployment automation.
Kubernetes 1.34 introduced several changes that are worth testing against .NET workloads before moving production clusters. Kubernetes follows a release lifecycle in which each minor release receives approximately 12 months of support, making upgrade planning an ongoing operational requirement.
The right strategy is not:
New Kubernetes Version
|
v
Upgrade Production
It is:
Current Cluster
|
v
Compatibility Tests
|
v
Application Tests
|
v
Infrastructure Tests
|
v
Staging Upgrade
|
v
Production Canary
|
v
Full Rollout
This article explains how to test .NET workloads against Kubernetes 1.34 before making the upgrade a production dependency.
Why Kubernetes Version Testing Matters
A .NET application does not run in isolation.
A typical production deployment may look like this:
Kubernetes Cluster
|
+------------------+------------------+
| | |
v v v
ASP.NET Core Worker Service CronJob
| | |
+------------------+------------------+
|
Service / Ingress
|
Config / Secrets
|
Storage / Network
The application may continue compiling perfectly while the Kubernetes environment introduces an operational regression.
Examples include:
Changed API behavior
Deprecated resources
Container runtime differences
Scheduling behavior
Resource management changes
Network policy issues
Admission-controller incompatibilities
Autoscaling differences
Monitoring-agent failures
That is why Kubernetes upgrades require infrastructure-level regression testing.
Understand the Kubernetes Release Lifecycle
Kubernetes minor releases have a relatively short support window compared with many application frameworks.
The Kubernetes project documents a release lifecycle of approximately one year for each minor release.
That means cluster upgrades should be treated as a recurring engineering process rather than an occasional emergency.
A healthy upgrade cycle looks like:
Current Version
|
v
Compatibility Review
|
v
Next Supported Version
|
v
Staging Validation
|
v
Production Upgrade
|
v
Next Upgrade Planning
Waiting until a cluster is close to the end of support creates unnecessary migration pressure.
Inventory the .NET Workloads First
Before testing Kubernetes 1.34, build an inventory.
For example:
| Workload | Type | Criticality | Dependencies |
|---|
| Orders API | ASP.NET Core | High | PostgreSQL, Redis |
| Worker | .NET Worker | High | Queue |
| Reporting API | ASP.NET Core | Medium | Database |
| Scheduler | CronJob | Medium | Storage |
| gRPC API | ASP.NET Core | High | Internal services |
Also record:
.NET version
Container base image
CPU requests
Memory requests
CPU limits
Memory limits
Replicas
Probes
Ingress
Service
ConfigMaps
Secrets
PersistentVolumes
HPA
PDB
NetworkPolicies
This inventory becomes your upgrade test matrix.
Verify the Container Images
Kubernetes upgrades do not automatically upgrade your .NET runtime.
Your container image remains responsible for the runtime environment.
For example:
FROM mcr.microsoft.com/dotnet/aspnet:8.0
WORKDIR /app
COPY publish/ .
ENTRYPOINT ["dotnet", "Orders.Api.dll"]
The important compatibility questions are:
Kubernetes 1.34
|
v
Container Runtime
|
v
Linux Kernel
|
v
.NET Runtime
|
v
ASP.NET Core
Test the complete chain.
Do not assume that because the .NET application works on the current cluster, it will automatically behave identically after the infrastructure upgrade.
Test Readiness and Liveness Probes
Health probes are one of the first things to validate.
A typical ASP.NET Core deployment might use:
livenessProbe:
httpGet:
path: /health/live
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /health/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 5
Test:
Application starts
|
v
Readiness becomes healthy
|
v
Traffic begins
|
v
Application failure
|
v
Liveness reacts
The upgrade should not change the expected health lifecycle.
Test Graceful Shutdown
.NET applications need to shut down cleanly when Kubernetes terminates a pod.
Test:
Pod receives termination
|
v
Application receives shutdown
|
v
Stop accepting work
|
v
Finish active requests
|
v
Release resources
|
v
Process exits
For APIs, monitor:
For workers:
Message acknowledgement
Job completion
Duplicate processing
Queue visibility timeout
A Kubernetes upgrade is a good opportunity to verify graceful termination instead of assuming it works.
Test Resource Requests and Limits
A common .NET deployment configuration is:
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "1Gi"
These values influence scheduling and runtime behavior.
Measure before and after the upgrade:
CPU utilization
Memory utilization
CPU throttling
OOMKills
Pod restarts
Request latency
Throughput
Do not simply compare whether the application is "up."
A workload can remain healthy while performance has degraded.
Benchmark ASP.NET Core APIs
Create a baseline before upgrading.
For example:
Endpoint RPS p95 p99
------------------------------------------------
GET /orders 1,200 42 ms 85 ms
GET /orders/{id} 1,500 31 ms 64 ms
POST /orders 850 73 ms 140 ms
After moving to Kubernetes 1.34, run the same workload.
The goal is not to prove that every number is identical.
The goal is to identify statistically meaningful regressions.
Track:
Throughput
p50 latency
p95 latency
p99 latency
Error rate
CPU
Memory
Pod restarts
Test Horizontal Pod Autoscaling
Autoscaling is especially important for .NET APIs.
A typical HPA might look like:
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: orders-api
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: orders-api
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
Test:
Low Load
|
v
2 Pods
High Load
|
v
Scale Out
Load Falls
|
v
Scale In
Measure:
Test Startup Time
Autoscaling only works well if new pods become ready quickly.
Measure:
Image Pull
+
Container Start
+
.NET Runtime Start
+
Application Initialization
+
Readiness
For a .NET service:
Pod Created
|
v
Container Started
|
v
.NET Starts
|
v
Dependency Initialization
|
v
Health Endpoint Ready
|
v
Traffic
Compare startup times before and after the cluster upgrade.
Test Deployment Rollouts
Use a controlled deployment:
kubectl rollout status deployment/orders-api
Then test:
Old Pods
|
v
New Pods
|
v
Readiness
|
v
Traffic Shift
|
v
Old Pods Terminated
Monitor:
Available replicas
Unavailable replicas
Restart count
HTTP errors
Latency
The upgrade should not introduce unexpected rollout behavior.
Test Pod Disruption Budgets
A production API may use:
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: orders-api
spec:
minAvailable: 2
selector:
matchLabels:
app: orders-api
Test the behavior during node maintenance and cluster operations.
The objective is:
Node Maintenance
|
v
Pod Eviction
|
v
Enough replicas remain available
A Kubernetes upgrade should not accidentally turn maintenance into an application outage.
Test Stateful .NET Services Carefully
Stateless APIs are generally easier to upgrade than stateful applications.
If your .NET workload depends on:
PostgreSQL
Redis
RabbitMQ
Kafka
PersistentVolume
test the dependency path separately.
For example:
ASP.NET Core
|
v
Database Service
|
v
Persistent Storage
Verify:
Test Kubernetes API Compatibility
One of the most important upgrade checks is API compatibility.
Search manifests for deprecated API versions:
grep -R "apiVersion:" ./k8s
Then review every resource used by the application.
Pay particular attention to:
Deployment
Ingress
CronJob
HorizontalPodAutoscaler
PodDisruptionBudget
NetworkPolicy
RBAC
The application code may be perfectly compatible while the Kubernetes manifests are not.
Test Ingress Behavior
For an ASP.NET Core API exposed through an ingress controller, test:
Client
|
v
Ingress
|
v
Service
|
v
ASP.NET Core Pod
Measure:
TLS termination
HTTP routing
Path rewriting
WebSockets
gRPC
Request body size
Timeouts
Client IP forwarding
For gRPC services, explicitly test HTTP/2 behavior.
Do not assume that because ordinary HTTP endpoints work, gRPC is also working correctly.
Test Network Policies
If NetworkPolicy is used, validate the application dependency graph.
For example:
orders-api
|
+----> PostgreSQL
|
+----> Redis
|
+----> payment-service
Test both:
Allowed connections
and:
Blocked connections
Security controls should remain effective after the upgrade.
Test Secrets and Configuration
.NET applications often depend on environment variables:
env:
- name: ConnectionStrings__Default
valueFrom:
secretKeyRef:
name: orders-secrets
key: database
Verify:
Secret exists
|
v
Pod receives secret
|
v
.NET configuration loads
|
v
Application connects
Also verify that secrets are not accidentally exposed through logs or diagnostic endpoints.
Test Background Workers
ASP.NET Core is not the only .NET workload that needs testing.
A worker service may look like:
public sealed class OrderWorker : BackgroundService
{
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
while (!stoppingToken.IsCancellationRequested)
{
await ProcessOrdersAsync(stoppingToken);
}
}
}
Test what happens when Kubernetes terminates the pod.
You want:
CancellationToken
|
v
Stop accepting new work
|
v
Finish current operation
|
v
Commit/Acknowledge
|
v
Exit
Otherwise, a rolling upgrade can cause duplicate jobs or lost work.
Test CronJobs
For scheduled .NET workloads, verify:
Schedule
Concurrency Policy
Job Completion
Failure Retry
History Limits
Pod Cleanup
A simple CronJob might run:
apiVersion: batch/v1
kind: CronJob
metadata:
name: daily-report
spec:
schedule: "0 2 * * *"
concurrencyPolicy: Forbid
Make sure the job still behaves correctly during node maintenance and upgrades.
Test Observability
A cluster upgrade should include an observability regression test.
Monitor:
Logs
Metrics
Traces
Events
Pod status
Node status
Application health
For .NET applications using OpenTelemetry, verify:
ASP.NET Core request
|
v
Trace
|
+----> Database span
|
+----> HTTP span
|
+----> Queue span
Do not treat observability as optional.
If telemetry disappears after an upgrade, diagnosing production issues becomes much harder.
Test Node-Level Behavior
The application runs on nodes, so test node characteristics as well.
Inspect:
kubectl get nodes
kubectl describe node <node-name>
Review:
Kubernetes version
Container runtime
CPU capacity
Memory capacity
Conditions
Taints
Labels
Your workloads may depend on node labels or affinity rules.
For example:
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: workload
operator: In
values:
- application
Verify that the scheduling constraints still produce the expected placement.
Build a Staging Cluster
The most valuable test is a production-like staging environment.
Use:
Same .NET version
Same container images
Same Helm charts
Same manifests
Same ingress
Same policies
Same observability
Same autoscaling
Only the Kubernetes version should intentionally differ.
Architecture:
Production
Kubernetes Current
|
| Same Workload
|
v
Staging
Kubernetes 1.34
This creates a meaningful comparison.
Use a Test Matrix
A useful upgrade matrix is:
| Area | Test | Pass Criteria |
|---|
| Build | Container starts | No startup failures |
| Health | Readiness | Healthy |
| API | Load test | Within threshold |
| Database | CRUD | No errors |
| Worker | Job processing | No lost/duplicate jobs |
| HPA | Scale test | Expected scaling |
| Ingress | HTTP/gRPC | Successful |
| Security | NetworkPolicy | Expected access |
| Storage | Volume | Read/write works |
| Observability | Logs/traces | Available |
| Deployment | Rollout | No outage |
This turns "Kubernetes upgrade testing" into measurable engineering work.
Run Failure Tests
Do not test only the happy path.
Test:
Pod crash
Node failure
Database unavailable
Network timeout
High CPU
High memory
Slow startup
Failed readiness
Failed liveness
Rolling restart
For example:
Database unavailable
|
v
ASP.NET Core
|
v
Connection Failure
|
v
Retry / Failure Handling
|
v
Recovery
The upgraded cluster should preserve the application's existing resilience behavior.
Canary the Production Upgrade
After staging passes, do not immediately upgrade every node.
Use a controlled rollout.
Production Cluster
|
v
Upgrade Small Node Group
|
v
Move Selected Workloads
|
v
Observe
|
v
Expand
Monitor:
Error rate
Latency
CPU
Memory
Pod restarts
Scheduling failures
Application logs
Kubernetes events
Only continue if the signals remain healthy.
Automate Upgrade Regression Tests
Manual testing does not scale.
A useful pipeline can be:
Infrastructure Change
|
v
Create Test Cluster
|
v
Deploy .NET Workloads
|
v
Run Integration Tests
|
v
Run Load Tests
|
v
Run Failure Tests
|
v
Collect Metrics
|
v
Pass / Fail
For application-level tests, standard .NET tooling can remain unchanged.
For example:
dotnet test
can validate application behavior while Kubernetes-specific checks validate deployment behavior.
Common Mistakes
Testing Only kubectl get pods
A green pod does not mean a healthy application.
Testing Only Startup
A workload can start correctly and fail under load.
Ignoring Autoscaling
Scaling behavior can be just as important as application correctness.
Ignoring Graceful Shutdown
Rolling upgrades exercise termination paths heavily.
Testing Only HTTP
gRPC, WebSockets, queues, and background workers need separate validation.
Skipping Observability
You need telemetry to understand upgrade behavior.
Upgrading Production First
Production should be the final validation environment, not the first.
Best Practices
Use a repeatable checklist:
[ ] Inventory .NET workloads
[ ] Inventory Kubernetes resources
[ ] Review deprecated APIs
[ ] Verify container images
[ ] Test readiness/liveness
[ ] Test graceful shutdown
[ ] Test resource limits
[ ] Benchmark APIs
[ ] Test HPA
[ ] Test deployments
[ ] Test PDB
[ ] Test ingress
[ ] Test gRPC
[ ] Test NetworkPolicy
[ ] Test secrets
[ ] Test background workers
[ ] Test CronJobs
[ ] Test observability
[ ] Test node scheduling
[ ] Run staging upgrade
[ ] Run failure tests
[ ] Run production canary
[ ] Monitor before full rollout
Advantages and Disadvantages
Advantages
Detects infrastructure regressions before production.
Protects .NET application availability.
Validates deployment manifests and APIs.
Provides measurable upgrade confidence.
Reduces emergency maintenance work.
Creates a repeatable Kubernetes upgrade process.
Disadvantages
Requires a representative staging environment.
Load testing infrastructure takes time to maintain.
Some infrastructure behavior is difficult to reproduce exactly.
Production canaries require careful operational planning.
Kubernetes upgrades can expose problems in third-party controllers and add-ons outside the application team's direct control.
Final Thoughts
Kubernetes upgrades should be treated as compatibility projects, not simply version changes.
For .NET teams, the application code is only one part of the upgrade surface. ASP.NET Core startup behavior, health probes, graceful shutdown, resource allocation, autoscaling, ingress, gRPC, background workers, storage, network policies, and observability all need to be validated.
Kubernetes' roughly one-year support lifecycle makes this especially important because waiting until maintenance pressure becomes urgent leaves less time for testing.
The safest approach is straightforward: establish a baseline on the current cluster, create a production-like Kubernetes 1.34 environment, run the same .NET workloads, measure application and infrastructure behavior, deliberately test failure scenarios, and then move through a controlled production canary.
The goal is not simply to prove that your .NET application runs on Kubernetes 1.34.
The goal is to prove that it still behaves correctly under production conditions after the upgrade.