Kubernetes 1.37: Pod-Level Resources vs Memory QoS

Memory problems in Kubernetes can be difficult to troubleshoot because memory does not behave like CPU.

A CPU limit can cause throttling when a workload tries to use more CPU than it is allowed to consume. Memory is different. When the system is under memory pressure, Kubernetes and the Linux kernel have to make decisions about which workloads can continue using memory and which workloads may need to be reclaimed or terminated.

This is where Memory QoS becomes important.

At the same time, Kubernetes has introduced a Pod-level resource model that lets developers describe resource requirements for a Pod rather than configuring everything only at the individual container level.

These two concepts are related, but they solve different problems.

Pod-level resources describe how much resource a Pod should have.

Memory QoS describes how memory pressure and memory protection can be handled.

Understanding the difference is important when running multi-container applications in production.

What Are Pod-Level Resources?

Traditionally, Kubernetes resource requests and limits are configured for individual containers.

For example:

resources:
  requests:
    cpu: "500m"
    memory: "256Mi"
  limits:
    cpu: "1"
    memory: "512Mi"

If a Pod contains two containers, each container can have its own resource configuration.

For example:

apiVersion: v1
kind: Pod
metadata:
  name: orders-api
spec:
  containers:
    - name: application
      image: orders-api:1.0
      resources:
        requests:
          cpu: "500m"
          memory: "256Mi"
        limits:
          cpu: "1"
          memory: "512Mi"

    - name: logging
      image: log-agent:1.0
      resources:
        requests:
          cpu: "100m"
          memory: "64Mi"
        limits:
          cpu: "200m"
          memory: "128Mi"

This model gives explicit control over each container.

A Pod-level resource model allows the resource configuration to represent the Pod as a whole.

Why Pod-Level Resources Matter

Consider a Pod containing:

orders-api
logging-agent

The application normally uses more memory than the logging agent.

But usage changes throughout the day.

For example:

Normal traffic

Application: 300Mi
Logging:      50Mi
Total:       350Mi

During a period of increased logging:

Application: 350Mi
Logging:     150Mi
Total:       500Mi

The resource requirement belongs to the combined workload, not necessarily to two completely independent applications.

Pod-level resources can provide a way to describe that combined requirement.

This is particularly useful for multi-container workloads where containers are tightly coupled.

What Is Memory QoS?

Memory Quality of Service, or Memory QoS, is about how memory resources are protected and managed when the system experiences memory pressure.

It builds on Linux memory-control mechanisms such as cgroups.

A simplified view looks like this:

Application
     |
     v
Container
     |
     v
Pod
     |
     v
Linux cgroup
     |
     v
Node memory

Kubernetes resource settings can influence how the workload is treated by the underlying system.

Memory QoS is therefore not another way of specifying the application's memory requirement.

It is about how those memory requirements are enforced and protected.

Pod-Level Resources and Memory QoS Solve Different Problems

This distinction is important.

Concept	Main Purpose
Pod-level resources	Describe resource requirements at the Pod level
Container-level resources	Describe resource requirements for individual containers
Memory QoS	Control or protect memory behavior under pressure
Memory request	Helps Kubernetes understand scheduling requirements
Memory limit	Places an upper boundary on memory usage under applicable conditions

Think about it this way:

Pod resources
    =
How much resource does this workload need?

Memory QoS
    =
How should memory be managed when pressure occurs?

They can work together, but they are not interchangeable.

Why Memory Is Different From CPU

Consider a CPU limit:

limits:
  cpu: "1"

If the process wants more CPU than its configured limit allows, CPU can be throttled.

Memory cannot be handled in exactly the same way.

A process cannot simply be given "slower memory."

If memory becomes unavailable, the system has to reclaim memory or take other action.

This is why memory configuration deserves particular attention.

A Simple Memory Example

Suppose a Pod has:

resources:
  requests:
    memory: "512Mi"
  limits:
    memory: "1Gi"

The application may normally use:

400Mi

During a traffic spike:

750Mi

And during an unusually large workload:

950Mi

These values are still below the configured limit.

But if the application keeps allocating memory and reaches:

1Gi

the workload is approaching the configured boundary.

If memory demand continues beyond what the system can provide, the container may eventually be terminated due to an out-of-memory condition.

Memory QoS and Requests

Memory requests are important because they provide information about the workload's expected resource needs.

For example:

requests:
  memory: "512Mi"

This tells Kubernetes that the workload requires that amount of memory for scheduling purposes.

It does not mean the application will always consume exactly 512Mi.

The actual usage can be lower or higher.

This is why resource requests should be based on measured application behavior.

Memory QoS and Limits

A memory limit is different:

limits:
  memory: "1Gi"

The limit establishes an upper memory boundary for the container or applicable Pod configuration.

Memory QoS can use the relationship between requests and limits when configuring the underlying memory controls.

The exact behavior depends on the Kubernetes version, feature configuration, container runtime, and operating system.

This is one reason teams should verify Memory QoS behavior in their own cluster rather than assuming that every cluster behaves identically.

A Multi-Container Example

Consider this Pod:

apiVersion: v1
kind: Pod
metadata:
  name: payment-service
spec:
  resources:
    requests:
      cpu: "1"
      memory: "768Mi"
    limits:
      cpu: "2"
      memory: "1.5Gi"

  containers:
    - name: application
      image: payment-service:1.0

    - name: telemetry
      image: telemetry-agent:1.0

The Pod has a combined resource configuration.

The application and telemetry containers are part of the same workload.

Now imagine their memory usage changes:

Scenario A

Application: 500Mi
Telemetry:    80Mi
Total:       580Mi

Later:

Scenario B

Application: 350Mi
Telemetry:   250Mi
Total:       600Mi

The resource requirement has shifted between containers while the combined usage remains within the Pod's budget.

This is one of the situations where Pod-level resource configuration can make sense.

Why Memory QoS Still Matters

Even with a Pod-level resource budget, the node can experience memory pressure.

For example:

Node memory
|
+-- Pod A
+-- Pod B
+-- Pod C
+-- System processes

If multiple workloads consume memory at the same time, the node may have less available memory than expected.

Kubernetes needs to manage these workloads according to their resource configuration and QoS characteristics.

This is why resource settings should always be considered at two levels:

Pod level
    +
Node level

A correctly configured Pod can still run on a node that is under severe memory pressure.

Checking Memory Usage

A quick way to check current Pod memory usage is:

kubectl top pod

For a namespace:

kubectl top pod -n production

For individual containers:

kubectl top pod <pod-name> --containers

Example:

POD             NAME          CPU(cores)   MEMORY(bytes)
payment-api     application   420m         620Mi
payment-api     telemetry      80m         120Mi

The total memory usage is roughly:

620Mi + 120Mi = 740Mi

If the Pod's configured memory limit is:

1.5Gi

the current usage is below that limit.

But a single snapshot does not tell you whether memory usage is stable.

Monitor Memory Over Time

Suppose monitoring shows:

09:00 -> 420Mi
10:00 -> 470Mi
11:00 -> 550Mi
12:00 -> 680Mi
13:00 -> 820Mi

That trend is more useful than one measurement.

It may indicate:

Increased traffic
Larger requests
A cache growing
A memory leak
Background processing
A configuration change

The next step should be application-level investigation rather than immediately increasing the memory limit.

Memory Pressure on the Node

You can inspect a node using:

kubectl describe node <node-name>

Look at the node conditions.

Memory pressure is particularly important.

If the node is under memory pressure, Kubernetes may need to evict Pods depending on the circumstances and their priority and QoS characteristics.

This means a Pod's resource configuration cannot be considered in isolation.

Quality of Service Classes

Kubernetes also has Pod QoS classes.

The commonly discussed classes are:

Guaranteed
Burstable
BestEffort

These are determined by the resource configuration of the containers in the Pod.

For example, a Pod where every container has matching CPU and memory requests and limits can qualify for the Guaranteed class under the applicable rules.

A Pod with some resource requests or limits but not matching requirements may be Burstable.

A Pod without resource requests and limits can fall into BestEffort.

You can check the QoS class with:

kubectl get pod <pod-name> -o jsonpath='{.status.qosClass}'

Example:

Burstable

This is useful when investigating memory pressure and eviction behavior.

Pod-Level Resources Do Not Automatically Mean Guaranteed QoS

This is an important distinction.

Adding a Pod-level resource configuration does not mean the Pod automatically receives the highest QoS classification.

QoS classification has its own rules.

If QoS behavior matters for your workload, inspect the actual Pod status:

kubectl get pod <pod-name> -o jsonpath='{.status.qosClass}'

Do not infer the QoS class just by looking at one resource field.

Memory QoS and Application Design

Kubernetes configuration cannot fix an application that continuously consumes memory without releasing it.

For example:

public class Cache
{
    private readonly Dictionary<string, byte[]> data = new();

    public void Add(string key, byte[] value)
    {
        data[key] = value;
    }
}

If this cache grows indefinitely, increasing the Pod memory limit only allows the application to consume more memory before hitting the problem.

The application may need:

Cache eviction
Size limits
Expiration
Better object lifetime management
Streaming instead of buffering
Reduced request sizes

Resource configuration is part of capacity management, not a replacement for application optimization.

Common Mistakes

Treating Memory Like CPU

Memory cannot simply be throttled when usage reaches a limit.

Setting Requests Too Low

If requests are much lower than actual requirements, scheduling and memory-pressure behavior may not match the workload's needs.

Setting Limits Too High

A very high memory limit can allow a workload to consume resources that other Pods need.

Ignoring the Node

A Pod may have available memory according to its own configuration while the node itself is under pressure.

Assuming QoS From One Setting

QoS classification depends on the overall resource configuration and Kubernetes rules.

Increasing Memory Without Finding the Cause

If memory usage is continuously increasing, investigate the application.

Best Practices

Measure Actual Memory Usage

Use metrics from realistic workloads.

Set Requests Based on Normal Requirements

The request should represent the workload's expected resource requirement rather than an arbitrary number.

Set Limits Based on Safe Maximum Usage

The limit should provide enough room for expected peaks while protecting the cluster.

Monitor Memory Trends

Look at historical data, not only current usage.

Check Container-Level Usage

A Pod can contain several containers with very different memory behavior.

Test Under Pressure

Load testing can reveal memory problems that are invisible during normal development workloads.

Troubleshooting OOMKilled Containers

If a container is repeatedly restarting, inspect the Pod:

kubectl describe pod <pod-name>

Check the container's state and last termination reason.

You can also use:

kubectl get pod <pod-name> -o jsonpath='{.status.containerStatuses[*].lastState}'

If you find an out-of-memory termination, investigate:

Application memory usage
Configured memory limit
Request size
Cache growth
Object allocation
Traffic pattern
Node memory pressure

Do not immediately increase the limit.

First determine whether the application is behaving normally.

A Practical Investigation

Suppose you have:

Memory request: 512Mi
Memory limit:   1Gi
Current usage:  850Mi

Start by checking historical usage.

If the application normally uses:

300–400Mi

and suddenly reaches:

850Mi

investigate what changed.

If the application normally uses:

800–900Mi

then the resource configuration may simply be too small for the actual workload.

The numbers need context.

Pod-Level Resources vs Memory QoS in Practice

It helps to keep the concepts separate:

Pod-Level Resources
        |
        v
Defines the workload's resource budget
        |
        v
Kubernetes scheduling and resource management

Memory QoS
        |
        v
Controls memory-related behavior
        |
        v
Linux memory management under pressure

They are complementary rather than competing features.

A team may configure Pod-level resources and still need to understand memory QoS, QoS classification, node pressure, and application memory behavior.

When Pod-Level Resources Are Useful

Consider Pod-level resources when:

Multiple containers form one logical workload.
Resource usage moves between those containers.
A combined resource budget is easier to manage.
The cluster supports the required Pod-level resource features.

When Container-Level Resources Are Better

Use explicit container-level resources when:

Containers have very different requirements.
One container must have strict resource boundaries.
Operators need clear per-container controls.
The containers perform independent workloads.

There is no requirement to use Pod-level resources simply because the Kubernetes version supports them.

Advantages and Disadvantages

Advantages	Disadvantages
Provides a resource view of the whole Pod	Can make per-container ownership less obvious
Useful for tightly coupled containers	Not required for every workload
Can support flexible resource sharing	Teams need to understand the interaction with container resources
Works alongside Kubernetes memory management	Does not solve application memory leaks
Useful for multi-container workloads	Node-level memory pressure still matters

Summary

Pod-level resources and Memory QoS address different parts of Kubernetes resource management.

Pod-level resources describe the CPU and memory requirements of the Pod as a combined workload. This can be useful when several containers work closely together and their resource usage changes over time.

Memory QoS is concerned with how memory is managed and protected when workloads and nodes experience memory pressure. It works with Kubernetes resource configuration and underlying Linux memory controls rather than replacing resource requests and limits.

For production workloads, the important thing is to look at the complete picture. Check Pod-level configuration, individual container usage, QoS classification, node memory pressure, and application memory behavior.

If a Pod is using too much memory, increasing the limit may be appropriate in some cases, but it should follow measurement and investigation. A growing cache, memory leak, unusually large request, or inefficient buffering may require an application-level fix instead.

The most reliable approach is to size resources from real workload data, monitor memory over time, and understand how Pod-level configuration and memory management work together.