Why cgroup v2 Matters for Kubernetes
A container can run out of memory even when a monitoring dashboard appears to show available RAM. A CPU-intensive application can also receive less processing time than developers expect, despite having a configured CPU limit.
These situations are easier to investigate when you understand how Kubernetes translates resource requests and limits into Linux kernel controls.
Kubernetes uses cgroups to isolate workloads and enforce resource consumption. The older cgroup v1 interface uses separate hierarchies for different controllers, while cgroup v2 provides a unified hierarchy with updated CPU, memory, and I/O controls.
Cgroup v2 has been stable in Kubernetes since version 1.25. Kubernetes now recommends it for Linux nodes, and cgroup v1 is deprecated starting with Kubernetes 1.35. Teams operating older clusters should plan their migration and verify that monitoring agents and application runtimes understand the newer interface.
The important point is that switching to cgroup v2 does not automatically make every application faster or change every resource limit. It changes the underlying controls and accounting interfaces that Kubernetes and supporting tools use.
How Kubernetes Resource Accounting Works
Kubernetes exposes CPU and memory requests and limits in a Pod specification.
CPU requests influence scheduling and the CPU share or weight assigned to a workload under contention.
CPU limits constrain how much CPU time a container can consume over a scheduling period.
Memory requests help the scheduler decide whether a node has enough capacity for a Pod.
Memory limits establish a boundary enforced by the kernel. Exceeding it can trigger memory reclamation or an out-of-memory (OOM) event.
Consider this container specification:
apiVersion: v1
kind: Pod
metadata:
name: resource-demo
spec:
containers:
- name: app
image: nginx:stable
resources:
requests:
cpu: "250m"
memory: "256Mi"
limits:
cpu: "1"
memory: "512Mi"
Here, the container requests 250 millicores of CPU and 256 MiB of memory. Its CPU limit is one CPU core, and its memory limit is 512 MiB.
These values serve different purposes. The scheduler uses requests to place the Pod, while the kernel enforces container resource controls. A memory request of 256 MiB does not mean that the container can never use more than that amount; the configured limit is 512 MiB.
Cgroup v2 provides the kernel interfaces used to enforce these boundaries, but the exact values written to cgroup files depend on the kubelet configuration, runtime, and operating system.
Identify Which cgroup Version a Node Uses
Before troubleshooting resource behavior, verify whether the node runs cgroup v1 or v2.
Run this command on the Linux node:
stat -fc %T /sys/fs/cgroup/
Interpret the output as follows:
Output | Meaning |
|---|
cgroup2fs
| The node uses cgroup v2 |
tmpfs
| The node uses the cgroup v1 filesystem layout |
You can also inspect the mounted control groups:
findmnt -t cgroup,cgroup2
On a cgroup v2 system, resource controllers share a unified hierarchy. Files such as cpu.max, cpu.stat, memory.current, and memory.max expose information about resource limits and consumption.
Do not assume that the same file paths or field names exist on cgroup v1 nodes. Monitoring agents and diagnostic scripts that read cgroup files directly must support the interface used by the host.
CPU Accounting: Shares Become Weights
Cgroup v1 and v2 use different interfaces for relative CPU allocation.
In cgroup v1, cpu.shares controls relative CPU weight under contention. In cgroup v2, cpu.weight provides the corresponding control, using a different numerical range and conversion behavior.
Cgroup v2 also exposes cpu.max, which represents a CPU bandwidth limit.
For example, a cpu.max value of:
100000 100000
means that the cgroup can consume up to 100,000 microseconds of CPU time in a 100,000-microsecond period. This corresponds to a quota of one CPU core, subject to the surrounding hierarchy and other constraints.
You can inspect the relevant files from the container's cgroup directory:
cat cpu.max
cat cpu.weight
cat cpu.stat
The output of cpu.stat includes counters such as the number of periods in which the cgroup was subject to CPU bandwidth control and the number of throttled periods. Depending on the kernel, it also reports the total throttled time.
These counters help distinguish a CPU limit problem from a general increase in application processing time.
For example, if request latency rises while throttling counters increase rapidly, the container may be reaching its configured CPU quota. If throttling remains low, investigate other causes such as locks, database waits, or insufficient concurrency.
A Note About Runtime Upgrades
The conversion from cgroup v1 CPU shares to cgroup v2 CPU weights has also received improvements in newer OCI runtimes. The conversion can affect how relative CPU priority is represented, particularly for small CPU requests.
This is a runtime-level behavior rather than a change to the meaning of Kubernetes CPU requests. After upgrading the container runtime, verify that monitoring tools and policy checks do not assume a fixed cpu.weight value for every request.
Memory Accounting: Inspect Actual Usage and Limits
Cgroup v2 provides several useful memory files.
File | Purpose |
|---|
memory.current
| Current memory charged to the cgroup |
memory.max
| Maximum memory permitted by the cgroup, or max if unlimited at that level |
memory.high
| A threshold that can trigger memory reclaim and throttling |
memory.events
| Counters for memory events, including high-threshold and OOM-related events |
memory.stat
| Breakdown of memory usage into categories such as anonymous memory and file-backed memory |
memory.peak
| Peak memory usage, where supported by the kernel |
These values are useful because a container's memory consumption includes more than the language runtime's heap.
Depending on the workload, memory charged to the cgroup can include anonymous allocations, page cache, and kernel memory. Consequently, a dashboard that only measures application heap usage may not explain why a container reaches its memory limit.
For a process-heavy or I/O-intensive application, compare the cgroup memory metrics with the application's own telemetry.
Example: Inspect Memory Usage
If you have access to the container's cgroup directory, inspect the current usage and limit:
cat memory.current
cat memory.max
cat memory.events
If memory.current reports a value close to memory.max, the container is approaching its configured memory boundary.
For example, a current usage of approximately 500 MiB against a 512 MiB limit indicates little remaining headroom. This does not by itself prove an OOM event will occur, but it is a reason to inspect allocation patterns, memory pressure, and recent event counters.
When troubleshooting a live Kubernetes workload, correlate these values with Pod events, container restart counts, and application logs. Avoid treating a single memory sample as proof of the root cause.
Why Monitoring Can Report Different Memory Values
A common source of confusion is that different monitoring layers measure different things.
A container's memory.current, the memory usage reported by a metrics pipeline, and the memory consumed by the application runtime are not necessarily identical.
For example, a Java application may report its heap separately from native memory. A container may also have memory charged for file-backed cache or kernel allocations that do not appear in heap metrics.
Kubernetes also has to decide how much node memory is considered available for eviction decisions. The treatment of active file-backed memory can affect those decisions, particularly for I/O-intensive workloads. Migrating to cgroup v2 does not automatically eliminate this behavior.
When a node reports memory pressure even though an application dashboard shows relatively low heap usage, investigate:
Container memory usage and limits.
memory.stat to understand the types of memory being charged.
Node-level memory pressure and available memory.
Pod eviction events and container OOM events.
The monitoring agent's version and its interpretation of cgroup metrics.
This approach helps identify whether the discrepancy comes from actual memory consumption, different accounting definitions, or a monitoring configuration issue.
Memory QoS and cgroup v2
Cgroup v2 also enables memory controls that are not available through the same interface in cgroup v1.
For example, memory.high can trigger reclaim and throttle a workload as it approaches a configured threshold. The memory.min and memory.low controls can provide different levels of memory protection when the relevant Kubernetes features and policies are enabled.
Kubernetes Memory QoS remains an alpha feature in the documented Kubernetes 1.36 behavior. Its availability and configuration should therefore be checked against the exact cluster version.
Do not enable alpha resource-management features in production without testing them under representative memory pressure. Incorrect reservation settings can leave insufficient memory for other workloads or system components.
Migration Checklist for Production Clusters
When migrating Linux nodes from cgroup v1 to cgroup v2, use a staged process.
Check operating system and kernel support. Cgroup v2 requires a compatible Linux distribution and kernel. Kubernetes documents Linux kernel 5.8 or later as a requirement.
Align the kubelet and runtime configuration. Both components must use compatible cgroup drivers. A mismatch can prevent workloads from starting correctly or cause resource-management problems.
Update monitoring agents. Verify support for cgroup v2 paths, counters, and memory statistics. For example, the Kubernetes documentation recommends cAdvisor 0.43.0 or later when using it as a standalone DaemonSet.
Check application runtimes. Confirm that Java, Node.js, and other runtimes detect container memory limits correctly. Older runtimes can incorrectly size heaps based on host memory.
Compare CPU behavior. Measure CPU throttling, utilization, latency, and throughput before and after migration.
Compare memory behavior. Review container memory usage, OOM events, evictions, and node-level pressure.
Test failure scenarios. Exercise CPU saturation and memory pressure in a non-production environment before moving critical workloads.
Plan the fallback carefully. Kubernetes versions and distributions differ in their cgroup v1 support. Confirm the supported fallback path for the cluster version before starting the migration.
Common Mistakes to Avoid
Assuming CPU requests are hard CPU limits.
Treating memory requests as the maximum memory a container can use.
Reading cgroup v1 files on a cgroup v2 node.
Assuming application heap usage equals total container memory usage.
Ignoring CPU throttling counters when investigating latency.
Upgrading the container runtime without checking CPU-weight conversion assumptions.
Enabling Memory QoS without testing its impact on node memory reservations.
Assuming that cgroup v2 automatically resolves every memory-pressure or eviction issue.
Conclusion
Cgroup v2 gives Kubernetes a unified resource-control interface and additional tools for CPU and memory management. For developers and platform engineers, the practical benefit is better visibility into the kernel controls that govern container behavior.
The most useful diagnostic files include cpu.max, cpu.stat, memory.current, memory.max, and memory.events. Together, they help explain CPU throttling, memory-limit pressure, and OOM-related behavior.
Before migrating a cluster, verify the cgroup version, align runtime configuration, update monitoring tools, and compare workload metrics under realistic conditions. Understanding the accounting model is more reliable than increasing resource limits whenever a dashboard looks unexpected.
Join the conversation! Your thoughts help the community grow.