Kubernetes 1.37 Watch Cache: API Server Recovery and etcd Performance

The Kubernetes API server is the central communication point for almost every Kubernetes component. Controllers, operators, command-line tools, and other clients continuously interact with the API server to read resources and receive change notifications.

When a Kubernetes cluster becomes large, the volume of API requests can become significant. Watch requests are particularly important because controllers use them to observe changes without repeatedly polling the API server.

Kubernetes 1.37 introduces improvements to the API server watch cache, including a new mechanism that allows the watch cache to recover more efficiently after an unexpected failure. These changes are designed to reduce unnecessary load on etcd and improve API server recovery behavior.

For developers and platform engineers running .NET workloads on Kubernetes, this matters because application deployments, scaling operations, configuration changes, and custom resources all depend on the Kubernetes control plane.

What Is the Kubernetes Watch Cache?

The Kubernetes API server uses a watch cache to efficiently serve watch requests.

A simplified request flow looks like this:

Controller
    |
    | Watch Pods
    v
API Server
    |
    | Watch Cache
    v
etcd

Without an effective cache, many clients would need to retrieve information directly from etcd or repeatedly query the API server.

The watch cache keeps recently observed resource state in memory and allows the API server to respond to watch requests without constantly replaying the same information from etcd.

This is particularly useful for resources such as:

Pods
Deployments
Services
ConfigMaps
Secrets
Nodes
Custom Resources

Why Watch Performance Matters

Kubernetes controllers are built around a reconciliation model.

A controller effectively follows this pattern:

Observe
   ↓
Detect change
   ↓
Reconcile desired state
   ↓
Update resource
   ↓
Observe next change

For example, when a Deployment changes from three replicas to five, Kubernetes components need to observe that change and react.

A simplified example is:

Deployment
    ↓
ReplicaSet
    ↓
Pods
    ↓
Scheduler / Kubelet

If API-server watch handling becomes inefficient, large clusters can experience increased control-plane load.

That does not necessarily mean the application itself is slow. The bottleneck can exist entirely in the Kubernetes control plane.

What Changed in Kubernetes 1.37?

Kubernetes 1.37 includes improvements to watch-cache initialization and recovery.

The important idea is that rebuilding a watch cache should not unnecessarily create additional pressure on etcd.

The Kubernetes API server can use the existing resource state more efficiently while reconstructing the cache instead of repeatedly performing expensive operations.

Conceptually:

Before
-----

Watch Cache Failure
       ↓
Rebuild Cache
       ↓
Heavy etcd Activity
       ↓
API Server Recovery


Improved Recovery
-----------------

Watch Cache Failure
       ↓
Recover / Reconstruct Cache
       ↓
Reduced unnecessary etcd work
       ↓
API Server resumes normal operation

The goal is not to eliminate etcd interaction. The goal is to make cache recovery more efficient and reduce unnecessary control-plane work.

Watch Cache and etcd

etcd is the backing key-value store used by Kubernetes for cluster state.

The relationship can be simplified as:

Clients
   |
   v
API Server
   |
   +--> Watch Cache
   |
   v
etcd

etcd stores the authoritative state.

The watch cache provides an API-server-side mechanism for efficiently serving watch operations.

This separation is important because an API-server problem does not automatically mean etcd is unhealthy.

Likewise, an etcd performance problem can cause API-server operations to become slow even when application Pods are healthy.

Understanding Watch Requests

A watch request asks the API server to continuously report changes to a resource.

For example, a client can request:

GET /api/v1/pods?watch=true

Instead of returning one static response, the API server keeps the connection open and sends events as resources change.

Conceptually:

{
  "type": "ADDED",
  "object": {
    "metadata": {
      "name": "orders-api"
    }
  }
}

A later change can generate:

{
  "type": "MODIFIED",
  "object": {
    "metadata": {
      "name": "orders-api"
    }
  }
}

Controllers can therefore react to changes without continuously polling the API server.

Why Recovery Behavior Matters

Watch caches can be rebuilt after events such as API-server restarts or other cache lifecycle events.

During recovery, the API server must reconstruct enough state to serve new watch requests correctly.

In a busy cluster, recovery can happen while other control-plane operations are still running.

This creates a potentially expensive scenario:

API Server Restart
       |
       +--> Controllers reconnect
       |
       +--> Watches reconnect
       |
       +--> Cache rebuild begins
       |
       +--> Normal API requests continue
       |
       +--> etcd receives additional traffic

A recovery mechanism that minimizes unnecessary work is therefore valuable for cluster stability.

Impact on .NET Workloads

Application developers may not interact with the watch cache directly, but Kubernetes workloads depend on the API server constantly.

For example, deploying a .NET application involves resources such as:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: orders-api
          image: example/orders-api:latest

Changing:

replicas: 3

to:

replicas: 5

requires Kubernetes control-plane components to observe and reconcile the desired state.

A healthy watch path helps those changes propagate normally.

The application code itself does not need to change because of watch-cache improvements.

How to Observe API Server Behavior

Kubernetes provides several useful commands when investigating control-plane behavior.

Check API-server Pods:

kubectl get pods -n kube-system

Check API-server logs:

kubectl logs -n kube-system \
  kube-apiserver-<node-name>

Inspect etcd health where applicable:

kubectl get --raw='/readyz?verbose'

The readiness endpoint provides detailed information about API-server readiness checks.

You can also check overall cluster behavior:

kubectl get --raw='/livez?verbose'

These endpoints are useful when distinguishing API-server availability problems from application-level failures.

Measuring API Request Performance

The Kubernetes API server exposes metrics that can help identify request latency and request volume.

For example, API-server monitoring can include:

apiserver_request_total
apiserver_request_duration_seconds

The exact metric names and available labels should be verified against the Kubernetes version and monitoring configuration being used.

When investigating performance, look for patterns such as:

Increasing API request latency
High request rates
Repeated watch reconnections
Elevated etcd latency
API-server restarts
Cache initialization or recovery activity

A single metric rarely explains the entire problem.

Troubleshooting API Server and Watch Problems

Step 1: Check API Server Availability

kubectl get --raw='/readyz?verbose'

If readiness checks are failing, inspect the API-server logs.

Step 2: Check API Server Restarts

kubectl get pods -n kube-system

Look at the restart count for the API-server Pods.

Step 3: Check etcd Health

If etcd is part of the suspected problem, inspect the etcd component and its available health information.

In a self-managed cluster, administrators can also inspect etcd logs and metrics.

Step 4: Look for Watch Reconnection Patterns

Controllers reconnecting repeatedly can increase API-server traffic.

If multiple components simultaneously reconnect after a control-plane event, the API server may experience a temporary request surge.

Step 5: Separate Control-Plane and Application Problems

If a .NET application's Pod is running normally but Kubernetes operations such as:

kubectl get pods

are slow, investigate the control plane before changing application resource settings.

Common Mistakes

Blaming etcd Immediately

API-server latency does not automatically mean etcd is the root cause.

The API server, watch cache, network, authentication, authorization, and etcd can all contribute to request latency.

Treating Watch as Polling

A watch is a long-lived event stream, not simply a repeated GET request.

Applications and controllers should use the appropriate Kubernetes client behavior instead of implementing inefficient polling.

Ignoring Control-Plane Metrics

Application dashboards often focus exclusively on CPU, memory, HTTP latency, and database performance.

For Kubernetes platforms, control-plane metrics are equally important.

Assuming Watch Cache Eliminates etcd Load

The watch cache reduces unnecessary work, but etcd remains the authoritative backing store for Kubernetes state.

Best Practices

Monitor API-server request latency and volume.
Monitor etcd health and latency separately.
Track API-server restarts.
Investigate repeated watch reconnections.
Keep Kubernetes controllers and operators updated according to supported versions.
Avoid unnecessary API polling in custom controllers or automation.
Use Kubernetes client libraries correctly for watch and informer-style workflows.
Separate application performance monitoring from control-plane monitoring.
Test control-plane recovery behavior in non-production environments.
Review Kubernetes 1.37 release documentation before changing production control-plane configurations.

Advantages and Disadvantages

Advantages

More efficient watch-cache recovery.
Can reduce unnecessary work against etcd during cache reconstruction.
Helps improve control-plane resilience after cache-related recovery events.
Benefits controllers that depend heavily on Kubernetes watches.
Requires no application changes for normal .NET workloads.

Disadvantages

Does not solve every API-server or etcd performance problem.
Requires control-plane monitoring to diagnose effectively.
Watch behavior can still be affected by network and API-server issues.
Large clusters can still generate substantial control-plane traffic.
Custom controllers can still create unnecessary API load through inefficient access patterns.

Practical Architecture for .NET Teams

A useful monitoring model for a Kubernetes-hosted .NET platform is:

                    Kubernetes Cluster
                           |
             +-------------+-------------+
             |                           |
       Application Layer           Control Plane
             |                           |
       .NET Metrics               API Server Metrics
       HTTP Metrics               Watch Metrics
       GC Metrics                 etcd Metrics
       Database Metrics           Recovery Events

This separation makes troubleshooting much easier.

For example, if an ASP.NET Core service reports normal request latency but deployment changes take unusually long, the investigation should move toward Kubernetes control-plane health rather than immediately profiling the .NET application.

Conclusion

The Kubernetes watch cache plays an important role in efficiently serving watch requests from the API server. Kubernetes 1.37 includes improvements around watch-cache recovery that aim to make cache reconstruction more efficient and reduce unnecessary pressure on etcd.

For .NET developers, the key takeaway is that application performance depends on more than the application container itself. Deployments, scaling, configuration updates, service discovery, and controllers all depend on a healthy Kubernetes control plane.

When troubleshooting Kubernetes performance, look at the complete path:

.NET Application
      ↓
Kubernetes Resources
      ↓
API Server
      ↓
Watch Cache
      ↓
etcd

Understanding these layers helps teams distinguish an application problem from a control-plane problem and makes Kubernetes incidents much easier to diagnose.