A Kubernetes application can run normally under light traffic and then experience unexpected throttling, memory pressure, or container restarts when the workload increases. Developers often investigate application code first, but the behavior may come from the Linux resource controls applied to the container.
Kubernetes relies on Linux control groups, commonly called cgroups, to manage and monitor container resources. The cgroup version used by a node affects how the kernel represents resource limits, reports usage, and enforces certain resource controls.
With cgroup v2, Linux uses a unified hierarchy for managing resources such as CPU and memory. This differs from cgroup v1, which uses separate hierarchies for different controllers. For Kubernetes operators, the distinction matters when diagnosing CPU throttling, memory limits, node resource pressure, and differences between container metrics and application behavior.
This article explains how cgroup v2 works with Kubernetes, how CPU and memory limits are enforced, how to inspect resource settings, and what developers should check when workloads behave differently after a node migration.
What Is cgroup v2?
A control group is a Linux kernel mechanism for organizing processes and controlling their resource consumption. Container runtimes use cgroups to apply resource constraints to container processes.
Linux supports two major cgroup implementations:
In cgroup v2, controllers are managed through a common hierarchy, usually mounted at /sys/fs/cgroup. Resource-specific files expose configuration and usage information.
For example, a cgroup v2 directory may contain files such as:
/sys/fs/cgroup/
cpu.max
cpu.stat
memory.max
memory.current
memory.events
These files provide information about CPU bandwidth limits, CPU statistics, memory limits, current memory usage, and memory events.
The exact files available depend on the controllers enabled and the cgroup hierarchy assigned to the workload.
How Kubernetes Uses cgroups
Kubernetes does not independently enforce CPU and memory limits inside each container. The kubelet, container runtime, and Linux kernel work together to configure and enforce the resource controls.
When a Pod specifies resource requests and limits, Kubernetes uses that information for scheduling and container configuration.
Consider the following Pod:
apiVersion: v1
kind: Pod
metadata:
name: api-service
spec:
containers:
- name: api
image: example/api:1.0
resources:
requests:
cpu: "500m"
memory: "256Mi"
limits:
cpu: "1"
memory: "512Mi"
The resource settings have different purposes.
requests.cpu indicates the CPU amount used for scheduling and related resource management decisions.
requests.memory indicates the memory amount used for scheduling.
limits.cpu defines the container's CPU usage ceiling through the configured CPU control mechanism.
limits.memory defines the container's memory limit.
The Pod requests half a CPU core and 256 MiB of memory, while its limits allow up to one CPU core and 512 MiB of memory.
A CPU request is not a guarantee that the container will always receive that amount of processing time. Likewise, a memory request is not a hard memory ceiling.
Kubernetes uses requests and limits differently, and cgroup v2 provides the kernel-level controls used to enforce applicable limits.
What Changes with cgroup v2?
The main difference is the resource-control interface exposed by Linux. Instead of separate controller hierarchies, cgroup v2 provides a unified hierarchy with resource-specific files.
For example, cgroup v2 commonly uses cpu.max for CPU bandwidth limits and memory.max for memory limits. These replace or differ from the corresponding interfaces used by cgroup v1.
For Kubernetes applications, the most important practical effects concern resource enforcement and observability.
Resource behavior | cgroup v1 | cgroup v2 |
|---|
CPU quota configuration | Commonly uses cpu.cfs_quota_us and cpu.cfs_period_us | Uses cpu.max |
CPU usage and throttling statistics | Uses controller-specific files such as cpu.stat | Uses cpu.stat with cgroup v2-specific statistics |
Memory limit | Commonly uses memory.limit_in_bytes | Uses memory.max |
Memory usage | Commonly uses memory.usage_in_bytes | Uses memory.current |
Memory event reporting | Uses controller-specific counters | Uses memory.events and related interfaces |
Resource hierarchy | Separate controller hierarchies | Unified hierarchy |
The filenames and metric semantics are important when building diagnostic scripts or monitoring agents that read cgroup information directly.
However, migrating to cgroup v2 does not automatically mean that every Kubernetes workload will become faster or consume less memory. Application performance still depends on the workload, kernel behavior, runtime configuration, and resource limits.
Understanding CPU Limits
Kubernetes CPU limits are typically enforced through CPU bandwidth controls. The kernel limits how much CPU time a container's processes can consume over a scheduling period.
In cgroup v2, the cpu.max file represents this quota and period.
For example:
100000 100000
The first number represents the CPU quota in microseconds, and the second represents the period in microseconds. This example permits 100,000 microseconds of CPU time per 100,000-microsecond period, corresponding to approximately one CPU core of bandwidth.
A container can have multiple runnable threads and still be constrained by the same aggregate CPU quota.
Why CPU throttling matters
Suppose an ASP.NET Core API has a CPU limit of one core. Under heavy concurrent traffic, several threads may become runnable at the same time. The container can exhaust its allowed CPU time for a scheduling period and be throttled until the next period.
This can increase request latency even when the application does not appear to have a high average CPU usage over a longer measurement interval.
To inspect CPU quota and throttling statistics on a cgroup v2 system, use:
cat /sys/fs/cgroup/cpu.max
cat /sys/fs/cgroup/cpu.stat
The paths above represent a cgroup directory. A container or Pod may use a nested cgroup, so locate the correct directory before interpreting the output.
In cpu.stat, fields such as nr_periods, nr_throttled, and throttled_usec can help reveal throttling activity when those statistics are available.
Do not assume that a nonzero throttling counter proves a performance problem. Compare the rate of throttling with request latency, throughput, and workload demand.
Understanding Memory Limits
Memory limits behave differently from CPU limits.
CPU throttling generally delays execution, while exceeding a container's memory limit can cause the kernel to invoke out-of-memory handling. Depending on the circumstances, a process in the container may be killed, causing the container to restart if its restart policy and Pod configuration require it.
In cgroup v2, memory.max represents the hard memory limit for a cgroup. memory.current reports the memory currently charged to it.
Inspect these files in the relevant cgroup:
cat /sys/fs/cgroup/memory.max
cat /sys/fs/cgroup/memory.current
cat /sys/fs/cgroup/memory.events
A numeric value in memory.max indicates the configured limit in bytes. The value max means that no finite limit is configured at that cgroup level.
The memory.events file contains counters for memory-related events, including limit and out-of-memory conditions. Interpret these counters alongside container termination reasons, Pod events, and application logs.
Why memory usage can be surprising
Memory reported by an application runtime may not exactly match the memory charged to its container cgroup.
For example, a .NET application may report managed heap usage while the container's memory accounting also includes other charged memory, such as native allocations and relevant file-backed memory.
Consequently, a process can approach its container memory limit even when its managed heap appears smaller than expected.
When investigating memory-related restarts, compare container memory metrics with runtime metrics and the kernel's memory event counters.
Check Which cgroup Version a Kubernetes Node Uses
Before diagnosing a resource problem, confirm which cgroup version the node uses.
On a Linux node with systemd, you can inspect the filesystem type mounted at /sys/fs/cgroup:
stat -fc %T /sys/fs/cgroup
On common systems, cgroup2fs indicates a cgroup v2 mount. A cgroup v1 system may expose a different filesystem type or separate controller mounts.
You can also inspect the node's kernel command line and runtime configuration when the filesystem layout is unclear.
Kubernetes node behavior depends on the operating system, kernel, container runtime, and kubelet configuration. Do not assume that all nodes in a cluster use the same cgroup version simply because they run the same Kubernetes release.
When a cluster uses mixed node configurations, compare workloads on equivalent nodes before attributing a performance difference to cgroup behavior.
Common Problems After a Migration
CPU throttling increases
If an application experiences higher latency after a node migration, compare its CPU limit, throttling counters, request latency, and throughput.
The correct response might be to adjust CPU limits, optimize CPU-intensive code, or reduce unnecessary concurrency. Increasing the CPU limit without measuring the workload can hide the actual bottleneck or create resource contention elsewhere.
Containers restart because of memory pressure
Inspect Pod status, termination reasons, node events, and memory metrics. An OOMKilled termination indicates that an out-of-memory event killed the container process, but additional investigation is needed to determine whether the cause was an unsuitable limit, a memory leak, or another allocation pattern.
Monitoring dashboards report different values
Monitoring agents may collect cgroup metrics directly. If a script expects cgroup v1 filenames, it may stop collecting data or report incomplete statistics after a node changes to cgroup v2.
Update the collection logic to support the correct hierarchy and metric semantics. Prefer established monitoring integrations that support both versions when the cluster needs to operate across different node configurations.
Resource behavior differs between nodes
Differences in kernel configuration, container runtime, node capacity, and Pod placement can affect observed behavior. Compare equivalent workloads with the same requests and limits before drawing conclusions about cgroup version alone.
Best Practices for Production Kubernetes Workloads
Set realistic requests and limits. Base them on observed CPU and memory consumption under representative workloads.
Monitor throttling and memory events. Correlate kernel-level resource metrics with application latency and container restarts.
Verify the node configuration. Confirm the cgroup version and runtime configuration before investigating resource-control differences.
Update monitoring integrations. Ensure scripts and agents understand the cgroup v2 hierarchy and files.
Test migrations under load. Compare CPU usage, throttling, memory pressure, and application behavior on equivalent nodes.
Avoid changing limits without evidence. A larger limit is not necessarily the right fix for inefficient queries, memory leaks, or excessive concurrency.
Account for runtime overhead. For .NET applications, monitor managed heap usage alongside total container memory.
Keep node configurations consistent where practical. Standardized operating-system and runtime settings make performance comparisons more reliable.
Advantages and Disadvantages
Advantages | Disadvantages |
|---|
Provides a unified hierarchy for resource controllers. | Existing scripts may rely on cgroup v1 paths. |
Exposes resource usage and event information through consistent interfaces. | Monitoring integrations may need changes. |
Supports CPU and memory enforcement at the kernel level. | Resource limits can still cause throttling or out-of-memory failures. |
Helps diagnose resource constraints when metrics are interpreted correctly. | Differences between runtime metrics and cgroup accounting can complicate troubleshooting. |
Provides a foundation for modern Linux resource management. | Actual behavior depends on kernel, runtime, and Kubernetes configuration. |
Summary
cgroup v2 changes how Linux exposes resource controls and statistics to Kubernetes workloads. The unified hierarchy uses interfaces such as cpu.max, cpu.stat, memory.max, and memory.current to describe CPU and memory constraints.
For developers and operators, the practical task is to understand how those controls affect application behavior. CPU limits can cause throttling, while memory limits can lead to out-of-memory termination. Monitoring tools must also use the correct cgroup interfaces to report accurate information.
When diagnosing problems, inspect the actual node configuration, compare cgroup metrics with application telemetry, and test changes under realistic load. This approach helps distinguish resource-limit problems from application inefficiencies and makes Kubernetes workloads easier to operate reliably.
Join the conversation! Your thoughts help the community grow.