A Kubernetes node can report healthy while an application behaves differently after a Linux upgrade. CPU usage may appear correct, memory pressure may increase, or a container that previously stayed within its resource limits may start getting terminated. These problems are particularly difficult to diagnose when the application code has not changed and the only significant difference is the node's operating system or cgroup configuration.

One possible cause is a migration from cgroup v1 to cgroup v2. Linux control groups, commonly called cgroups, provide the kernel mechanisms that container runtimes use to account for and restrict resources such as CPU and memory. Kubernetes relies on these mechanisms to enforce container resource settings and manage node workloads.

Moving to cgroup v2 changes the underlying resource-control interface. Although Kubernetes and modern container runtimes support cgroup v2, a successful migration still depends on compatible software versions, correct node configuration, and workload testing.

The main task for cluster operators is to verify that CPU scheduling, memory enforcement, resource accounting, and monitoring continue to behave as expected after the migration.

What Is cgroup v2?

Linux control groups allow the operating system to organize processes into groups and apply resource-management rules to them. Containers use these mechanisms to isolate workloads and limit their consumption of shared node resources.

For example, a container may have a CPU limit that constrains how much processing time it can consume. A memory limit can restrict its memory usage and influence what happens when that usage exceeds the configured boundary.

The cgroup v1 implementation uses separate controller hierarchies. CPU, memory, and other controllers can be organized through different hierarchies, which makes the resource-management structure more complicated.

cgroup v2 introduces a unified hierarchy and a different interface for resource controllers. It provides a more consistent model for managing resources, but it also changes the files, controller behavior, and kernel interfaces used by container runtimes.

For Kubernetes, the important point is that cgroup v2 is not simply a different filesystem layout. It affects the mechanism through which the node enforces container resource settings.

Why Kubernetes Operators Need to Check Compatibility

A migration can involve several components:

These components must work together. Updating the operating system while leaving an incompatible runtime or monitoring agent in place can create problems that appear to originate in Kubernetes even though the underlying issue is a mismatch between components.

The kubelet's cgroup driver must also be compatible with the container runtime's configuration. A mismatch can prevent pods from starting correctly or produce unexpected resource-management behavior.

Modern Kubernetes environments generally use the systemd cgroup driver when systemd manages the node's cgroups. The important requirement is consistency between the kubelet and runtime, not blindly changing the driver without understanding the existing environment.

Before planning the migration, establish the versions and configuration of every relevant component. Do not assume that a node is using cgroup v2 merely because it runs a recent Linux distribution.

Step 1: Confirm Which cgroup Version the Node Uses

Begin by checking the cgroup filesystem mounted on the node.

On Linux, the following command provides a useful initial check:

stat -fc %T /sys/fs/cgroup/

A result of cgroup2fs indicates that the path is mounted as a unified cgroup v2 hierarchy. A cgroup v1 environment generally uses controller-specific hierarchies instead.

You can also inspect the process cgroup information:

cat /proc/self/cgroup

On a unified cgroup v2 system, the output commonly contains a line resembling:

0::/user.slice/...

These commands are diagnostic indicators, not a complete compatibility assessment. Containers, nested environments, and custom operating-system configurations can affect what a process sees.

Next, check the Kubernetes node and runtime information:

kubectl get nodes -o wide
kubectl describe node NODE_NAME

Replace NODE_NAME with the relevant node name. Review the reported operating-system image, kernel version, container runtime, node conditions, and recent events.

The Kubernetes API does not expose every low-level cgroup detail, so node-level inspection may still be necessary. On managed Kubernetes services, use the provider's supported diagnostics and configuration mechanisms rather than assuming unrestricted access to the host.

Step 2: Verify the cgroup Driver

The cgroup driver determines how the kubelet and container runtime organize and manage container cgroups.

A common configuration uses the systemd driver when systemd manages the node. The kubelet and runtime should use compatible driver settings.

On nodes where the container runtime is managed directly, inspect its configuration using the appropriate runtime-specific tools. For containerd, the relevant setting depends on the installed version and configuration format. Avoid copying configuration fragments from another version without checking the supported schema.

For the kubelet, inspect the configuration used by the actual service rather than relying only on a local configuration file that may not be active.

The key questions are:

  1. Which cgroup driver does the kubelet use?

  2. Which cgroup driver does the container runtime use?

  3. Does the node's operating system manage cgroups through systemd?

  4. Are the installed Kubernetes and runtime versions compatible with the selected configuration?

A migration should not proceed until these settings are understood. Changing the cgroup driver on an established node can affect how workloads are managed, so the change requires a controlled procedure and a clear recovery plan.

Step 3: Check CPU Resource Enforcement

CPU requests and limits are frequently misunderstood during cgroup migrations because they serve different purposes.

A CPU request contributes to scheduling decisions and can influence relative CPU allocation under contention. A CPU limit constrains the CPU time available to a container through kernel resource controls.

In cgroup v2, CPU controls use interfaces such as cpu.max and cpu.weight. Their values are not direct replacements for the cgroup v1 interface files, so monitoring agents and diagnostic scripts that inspect the old paths may stop working or report misleading values.

For example, a cgroup v2 cpu.max value may look like this:

200000 100000

This represents a quota of 200,000 microseconds over a period of 100,000 microseconds, corresponding to an average CPU allowance of two CPU cores for that cgroup.

This is an illustrative example of the Linux interface, not a recommended limit for every workload. The effective behavior depends on the cgroup to which the setting applies and the resource controls configured by the runtime.

After migration, validate CPU behavior using representative workloads. Compare throttling metrics, application latency, CPU utilization, and throughput with a baseline from the previous environment.

If a service becomes slower, do not immediately increase its CPU limit. First determine whether it is being throttled, waiting on another dependency, or affected by an unrelated change in workload or node capacity.

Step 4: Check Memory Limits and OOM Behavior

Memory behavior deserves equal attention because a container can run normally during light testing and fail only when its memory consumption increases.

cgroup v2 uses files such as memory.current, memory.max, and memory.events to expose memory usage, limits, and selected events for a cgroup. These interfaces differ from those used by cgroup v1, which affects scripts and monitoring integrations that expect the older hierarchy.

The memory.max file represents the hard memory limit for a cgroup when configured. If the workload exceeds the applicable limit and the kernel cannot reclaim enough memory, memory allocation can fail or processes may be terminated through the out-of-memory mechanism.

Kubernetes operators should verify that container memory limits continue to be enforced as intended and that memory-related events are visible in monitoring.

Use representative workloads to test normal operation and controlled memory pressure. Observe container restarts, termination reasons, node memory pressure, and application-level errors.

Do not use an uncontrolled memory-exhaustion test on a production node. A test intended to verify one container's limit can affect other workloads if the test is incorrectly configured or the node has insufficient spare capacity.

Step 5: Validate Monitoring and Observability

One of the most easily overlooked migration problems is a monitoring agent that still expects cgroup v1 file paths.

A dashboard may show missing CPU metrics, zero memory usage, or an unexpected gap in container statistics even though the application itself is running. In other cases, the metrics remain available but change meaning or aggregation behavior.

Review the versions and compatibility documentation for the node exporter, container metrics collectors, log agents, and any custom scripts that read directly from /sys/fs/cgroup.

Where possible, use supported monitoring interfaces rather than hard-coding paths into application code. If direct cgroup inspection is necessary, make the implementation aware of the cgroup version and validate the interpretation of each metric.

After the migration, compare monitoring output with independent node-level measurements. A dashboard displaying no memory usage should not be treated as evidence that the node has no memory pressure.

Observability must be validated as part of the migration, because an environment that cannot report resource problems accurately is harder to operate safely.

Step 6: Test Workloads Before Migrating the Entire Cluster

A node-level migration should begin with a representative test environment or a small group of non-critical nodes.

Choose workloads that exercise different resource behaviors: a CPU-intensive service, a memory-intensive application, a service with strict latency requirements, and a background worker with variable load.

Compare their behavior before and after the migration. Relevant measurements include:

Testing should include realistic concurrency. A workload that behaves correctly in isolation may behave differently when several containers compete for CPU and memory on the same node.

If the cluster uses custom admission policies, resource quotas, or monitoring rules, include those controls in the validation process. The migration is not complete merely because the container runtime starts and the kubelet reports a healthy node.

Common Migration Mistakes

Assuming the operating-system upgrade completes the migration automatically. A newer distribution may default to cgroup v2, but runtime compatibility, cgroup drivers, and monitoring components still need verification.

Changing the cgroup driver without checking both components. A kubelet and container runtime configured inconsistently can cause workload startup and management problems. Confirm the active configuration before making changes.

Using old monitoring paths on a cgroup v2 node. Scripts that read cgroup v1 files may stop collecting metrics or interpret resource values incorrectly. Update the collector and validate the resulting measurements.

Treating CPU requests and limits as equivalent. Requests influence scheduling, while limits constrain runtime resource consumption. Test both scheduling behavior and actual CPU enforcement.

Migrating every node at once. A cluster-wide change increases the impact of an overlooked compatibility issue. Use a staged rollout with health checks and a defined rollback procedure.

Advantages and Disadvantages

Advantages

Disadvantages

Summary

Migrating Kubernetes nodes from cgroup v1 to cgroup v2 requires more than confirming that the new hierarchy is mounted. Operators should verify the kubelet and runtime versions, check cgroup driver consistency, test CPU and memory enforcement, and confirm that monitoring continues to report accurate resource usage.

The most reliable approach is a staged migration supported by representative workload tests and a clear rollback plan. Pay particular attention to CPU throttling, memory-limit enforcement, OOM behavior, and monitoring agents that inspect cgroup files directly.

cgroup v2 provides a modern Linux resource-management model, but successful adoption depends on validating the entire node stack. The goal is to preserve predictable workload behavior while moving to the new resource-control interface.