Kubernetes  

Kubernetes 1.37 Memory QoS: Improving .NET Container Performance Under Memory Pressure

Memory pressure is one of the most common causes of instability in containerized applications. A .NET application may work correctly during normal traffic but become slow, restart unexpectedly, or get terminated when several workloads compete for the same node memory.

Kubernetes provides memory requests and limits to help control resource allocation, but memory pressure can still create difficult situations. Kubernetes 1.37 continues improving resource management with MemoryQoS, which uses Linux memory control mechanisms to apply stronger memory protection and throttling behavior to containers.

For .NET workloads, understanding how Kubernetes and the Linux kernel manage memory is important because the .NET runtime, garbage collector, native libraries, and application dependencies all consume memory inside the container.

What Is Kubernetes MemoryQoS?

MemoryQoS is designed to provide more predictable memory behavior by using Linux cgroup memory controls.

A typical Kubernetes container might define:

resources:
  requests:
    memory: "256Mi"
  limits:
    memory: "512Mi"

The request tells Kubernetes how much memory the Pod needs for scheduling purposes.

The limit defines the maximum memory the container is allowed to consume before the kernel may terminate processes because of memory pressure.

MemoryQoS adds additional controls around how memory pressure is handled rather than relying only on a hard memory limit.

This matters because memory is different from CPU. CPU can generally be throttled, while excessive memory consumption can eventually result in an out-of-memory condition.

Why Memory Pressure Matters for .NET

Consider an ASP.NET Core application running inside a container:

ASP.NET Core
     |
     +-- Managed heap
     +-- Thread stacks
     +-- Native allocations
     +-- JIT/runtime memory
     +-- Libraries
     +-- Buffers
     +-- Database/network resources

The memory limit applies to the container's overall memory usage, not simply the managed .NET heap.

Therefore, this assumption can be dangerous:

Container limit = 512 MiB
.NET GC heap = 400 MiB
Therefore everything is safe

The process can consume additional memory outside the managed heap.

A container reaching its memory limit can therefore experience an OOM kill even when application-level heap metrics do not appear to explain the entire memory usage.

Memory Requests vs Memory Limits

The first step is understanding the difference between requests and limits.

SettingPurposeUsed By
Memory requestRepresents expected resource requirementKubernetes scheduler
Memory limitDefines container memory boundaryLinux cgroups / kubelet
MemoryQoS controlsInfluences memory-pressure behaviorLinux kernel
.NET GC settingsControls managed memory behavior.NET runtime

A production Deployment might use:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: orders-api
spec:
  replicas: 3
  selector:
    matchLabels:
      app: orders-api
  template:
    metadata:
      labels:
        app: orders-api
    spec:
      containers:
        - name: orders-api
          image: myregistry/orders-api:latest
          resources:
            requests:
              cpu: "250m"
              memory: "256Mi"
            limits:
              cpu: "1"
              memory: "512Mi"

The important point is that resource values should be based on observed workload behavior rather than arbitrary defaults.

How MemoryQoS Helps

MemoryQoS uses Linux cgroup mechanisms such as memory.min and memory.high to provide memory protection and throttling behavior.

Conceptually:

                    Node Memory
                         |
             +-----------+-----------+
             |                       |
        .NET Pod A              .NET Pod B
             |                       |
       memory.min              memory.min
       memory.high             memory.high
             |                       |
       Memory pressure       Memory pressure
             |                       |
       Controlled reclaim    Controlled reclaim

The goal is to make memory pressure more predictable before the system reaches an unrecoverable out-of-memory condition.

However, MemoryQoS does not mean a container can exceed its configured memory limit indefinitely. The hard limit remains an important boundary.

Understanding memory.min

memory.min provides a level of memory protection.

When memory pressure occurs, memory protected by memory.min is treated differently from memory that does not have the same protection.

Kubernetes can derive these values from Pod resource requests when MemoryQoS is enabled.

For example:

Pod memory request
       ↓
memory.min
       ↓
Memory protection

This makes resource requests more meaningful than simply being scheduling information.

A realistic memory request therefore helps Kubernetes determine which workloads should receive greater protection when memory becomes scarce.

Understanding memory.high

memory.high provides a threshold at which memory usage can be throttled rather than immediately triggering an OOM kill.

Conceptually:

Memory usage
     |
     |       memory limit
     |-------------------------
     |
     |       memory.high
     |-------------------------
     |
     |       Normal operation
     |
     +---------------------------->

Crossing the high threshold can cause reclaim and throttling behavior.

This gives the system an opportunity to slow down memory growth before the hard limit is reached.

For applications with temporary memory spikes, this can be preferable to immediately reaching the container's maximum memory boundary.

MemoryQoS and the .NET Garbage Collector

The .NET garbage collector manages managed objects, but Kubernetes manages the container's overall memory environment.

For example:

public async Task<byte[]> DownloadReportAsync(
    CancellationToken cancellationToken)
{
    using var stream = new MemoryStream();

    await _httpClient.GetStreamAsync(
        "/reports/monthly",
        cancellationToken)
        .ContinueWith(async task =>
        {
            await (await task).CopyToAsync(
                stream,
                cancellationToken);
        });

    return stream.ToArray();
}

Code that unnecessarily loads a large payload into memory can increase pressure on the managed heap.

A more memory-conscious design is to process data as a stream rather than creating multiple large in-memory copies.

For example:

public async Task SaveReportAsync(
    Stream destination,
    CancellationToken cancellationToken)
{
    await using var source =
        await _httpClient.GetStreamAsync(
            "/reports/monthly",
            cancellationToken);

    await source.CopyToAsync(
        destination,
        cancellationToken);
}

The exact memory behavior depends on the application and underlying libraries, but avoiding unnecessary large allocations is generally important for containerized workloads.

Do Not Treat Kubernetes MemoryQoS as a .NET GC Setting

MemoryQoS does not replace .NET runtime configuration.

These are separate layers:

Kubernetes
    ↓
Container cgroup memory controls
    ↓
Operating system
    ↓
.NET runtime
    ↓
Garbage Collector
    ↓
Application allocations

The application team should therefore monitor both infrastructure-level and application-level memory metrics.

Useful .NET metrics include:

  • GC heap size

  • Allocation rate

  • Gen 0 collections

  • Gen 1 collections

  • Gen 2 collections

  • Working set

  • Process memory

Kubernetes-level monitoring should include:

  • Container working set

  • Container memory usage

  • Memory requests

  • Memory limits

  • Pod restarts

  • OOM kill events

  • Node memory pressure

Checking Memory Usage in Kubernetes

Start with:

kubectl top pods

For a specific Pod:

kubectl top pod orders-api-7c8f7b9d5f-abc12

Inspect the Pod configuration:

kubectl describe pod orders-api-7c8f7b9d5f-abc12

Look for termination information:

Reason: OOMKilled
Exit Code: 137

An OOMKilled status indicates that the container was terminated because of an out-of-memory condition.

Do not immediately assume that the .NET garbage collector is the problem. First determine whether the container exceeded its cgroup memory boundary and then investigate which component consumed the memory.

Common Causes of High Memory Usage in .NET

Large Object Allocations

Creating large arrays, strings, or buffers can rapidly increase memory usage.

For example:

var data = new byte[100 * 1024 * 1024];

This allocates approximately 100 MiB for a single array.

Repeated allocations of this scale can quickly create memory pressure.

Unbounded Caching

A cache without an appropriate size or expiration policy can continuously grow.

For example:

_memoryCache.Set(
    cacheKey,
    largeObject);

A production cache should have an intentional eviction strategy when cached objects can consume significant memory.

Loading Entire Results

Avoid loading unnecessarily large database results into memory.

Instead of retrieving an entire dataset:

var orders = await db.Orders
    .ToListAsync(cancellationToken);

consider pagination or streaming when the application's requirements allow it.

Memory Leaks Through Long-Lived References

Managed memory can remain reachable even when the application no longer logically needs it.

Static collections and incorrectly scoped services are common areas worth investigating.

Troubleshooting Memory Pressure

When a .NET Pod experiences memory problems, follow a structured process.

Step 1: Check Pod Status

kubectl get pods

Look for repeated restarts.

Step 2: Inspect Previous Container Logs

kubectl logs orders-api-7c8f7b9d5f-abc12 --previous

This can reveal application errors immediately before termination.

Step 3: Check Resource Configuration

kubectl describe pod orders-api-7c8f7b9d5f-abc12

Verify the configured memory request and limit.

Step 4: Compare Application and Container Metrics

If the .NET GC heap appears stable but container memory keeps increasing, investigate native allocations, buffers, libraries, and other process memory.

Step 5: Check Node Pressure

kubectl describe node <node-name>

Look for memory-pressure conditions.

This helps distinguish a container-specific problem from broader node-level memory pressure.

Common Mistakes

Setting Very Low Memory Limits

A memory limit that is too low can cause unnecessary OOM kills.

Setting Very High Limits Without Requests

Large limits do not guarantee that the Pod will receive sufficient memory during scheduling or node pressure.

Requests and limits should be considered together.

Monitoring Only GC Heap

The managed heap is only one part of total process memory.

Assuming MemoryQoS Prevents OOM Kills

MemoryQoS improves memory-pressure handling; it does not eliminate the possibility of an OOM condition.

Ignoring Container Restarts

Repeated restarts can be an early indication of memory instability.

Best Practices

  1. Measure the application's real memory usage before setting requests and limits.

  2. Monitor both .NET GC metrics and container memory metrics.

  3. Avoid unnecessary large in-memory buffers.

  4. Use streaming and pagination for large datasets where appropriate.

  5. Configure bounded caches.

  6. Investigate OOMKilled events rather than simply increasing memory limits.

  7. Keep requests realistic so scheduling and memory protection reflect actual workload requirements.

  8. Test applications under realistic concurrent workloads.

  9. Monitor node-level memory pressure.

  10. Review runtime, container, and Kubernetes behavior together when diagnosing memory issues.

Advantages and Disadvantages

Advantages

  • Provides additional memory-pressure controls.

  • Uses Linux cgroup mechanisms for memory protection and throttling.

  • Can make resource requests more meaningful for memory protection.

  • Helps workloads behave more predictably under contention.

  • Complements application-level memory management.

Disadvantages

  • Does not eliminate OOM kills.

  • Requires understanding Kubernetes and Linux memory behavior.

  • Incorrect requests can result in inappropriate protection or scheduling.

  • Throttling under memory pressure can affect application performance.

  • Application-level memory problems still require application-level fixes.

Conclusion

Kubernetes MemoryQoS provides an additional layer of memory management for containerized workloads. For .NET applications, it is particularly important to understand that Kubernetes controls the container's overall memory while the .NET garbage collector manages only the runtime's managed memory.

A stable production configuration therefore requires more than choosing a memory limit.

Teams should measure actual application behavior, configure realistic requests and limits, monitor GC and container memory independently, and investigate OOM events systematically.

MemoryQoS can improve behavior under memory pressure, but it should be viewed as part of a broader resource-management strategy rather than a replacement for fixing memory-heavy application code.