A chatbot can respond quickly when a conversation contains only a few messages.
The same chatbot can become noticeably slower after a conversation grows to hundreds of messages.
The model has not necessarily become slower. The problem is that the amount of context being processed and retained during inference has increased.
Large language models use a structure called the key-value cache, commonly called the KV cache, to avoid recomputing attention information for tokens that have already been processed. During generation, the model reuses this cached information instead of starting the attention calculation from scratch for every new token.
This improves performance, but the cache consumes GPU memory.
For a short request, that may not matter much.
For many long conversations running concurrently, KV-cache memory can become one of the main limits on an inference server.
A simplified inference request looks like this:
User message
|
v
Conversation history
|
v
Tokenizer
|
v
Model
|
+---- KV Cache
|
v
Generated tokens
As the conversation grows, the amount of context grows.
As concurrent conversations grow, the total KV-cache requirement grows as well.
This creates a production problem:
More conversation history
|
v
More KV cache
|
v
Less available GPU memory
|
v
Fewer concurrent requests
|
v
Longer queues
|
v
Higher latency
The problem becomes even more important in Kubernetes because adding more application replicas does not automatically solve GPU-memory pressure inside an inference worker.
The routing layer, model server, GPU scheduler, and application all have to work together.
What the KV Cache Stores
During transformer inference, attention uses key and value tensors.
For a sequence of tokens, the model generates these values as it processes the input.
Instead of recalculating them for every generated token, an inference engine can keep them in memory.
That retained information is the KV cache.
Conceptually:
Token 1 ---> Key / Value
Token 2 ---> Key / Value
Token 3 ---> Key / Value
Token 4 ---> Key / Value
|
v
KV Cache
When the next token is generated, the model can reuse the cached key and value information.
Without caching, the model would repeatedly perform work for previously processed tokens.
With caching, the server spends additional GPU memory to reduce repeated computation.
This creates a basic tradeoff:
More cache
|
+-- Less repeated computation
|
+-- More GPU memory usage
The cache therefore improves inference efficiency but is not free.
Why Long Conversations Are Different
Consider a chatbot with this conversation:
User: Explain Kubernetes.
Assistant: ...
User: What about Pods?
Assistant: ...
User: Explain Services.
Assistant: ...
The next request may include the previous conversation as context.
If the conversation becomes very long:
System instructions
+
Conversation history
+
Current user message
the input sequence becomes larger.
A model serving system may maintain KV-cache state for the active sequence while generating the response.
Now imagine 100 users doing the same thing concurrently.
The problem is no longer one large conversation.
It is many large conversations competing for the same GPU memory.
GPU
|
+-- Conversation A -> large KV cache
+-- Conversation B -> large KV cache
+-- Conversation C -> large KV cache
+-- Conversation D -> large KV cache
+-- ...
Eventually, the GPU becomes the limiting resource.
Why GPU Utilization Can Be Misleading
A common monitoring mistake is to look only at GPU utilization.
For example:
GPU utilization: 70%
GPU memory: 96%
Queue length: 15
The GPU is not fully saturated from a compute-utilization perspective, but the inference server is already under memory pressure.
The opposite situation can also happen:
GPU utilization: 95%
GPU memory: 55%
Queue length: 2
Here the bottleneck may be compute rather than memory.
This distinction matters because the solution is different.
If compute is the bottleneck, adding capacity or improving batching may help.
If KV-cache memory is the bottleneck, adding more concurrency to the same GPU can make the situation worse.
The First Fix: Stop Sending Unnecessary History
The easiest way to reduce KV-cache pressure is to reduce the amount of context sent to the model.
Many applications simply send the complete conversation history:
System prompt
Message 1
Message 2
Message 3
...
Message 200
Current message
This is easy to implement, but it is not always necessary.
Instead, the application can maintain a shorter working context:
System prompt
Conversation summary
Recent messages
Current message
For example:
Original conversation:
200 messages
Model context:
Summary of older messages
+
Last 20 messages
+
Current request
The application still preserves the important information while reducing the amount of active context.
This is one of the most effective architectural controls because it reduces the problem before the request reaches the inference server.
Use Conversation Summarization Carefully
Summarization can reduce context size, but it introduces another model operation.
A typical workflow looks like this:
Conversation grows
|
v
Context exceeds threshold
|
v
Summarize older messages
|
v
Store summary
|
v
Keep recent messages
The next request becomes:
System prompt
+
Stored summary
+
Recent conversation
+
New message
The summary should preserve information that affects future answers.
For example:
User is building a .NET application.
The application uses PostgreSQL.
Authentication is handled by Microsoft Entra ID.
The user prefers REST APIs.
It should not preserve every sentence from the original conversation.
The objective is to preserve useful state, not the entire transcript.
Do Not Summarize Every Request
Running a summarization model after every message can introduce unnecessary cost and latency.
A better strategy is to trigger summarization based on a threshold.
For example:
Context < threshold
|
v
Continue normally
Context >= threshold
|
v
Summarize older messages
|
v
Continue
The threshold should be based on the model's context window and the application's actual latency and memory requirements.
Do not wait until the model reaches the absolute maximum context size.
Leave room for the new user message and the expected generated output.
Keep Input and Output Budgets Separate
A request can contain:
Input tokens
+
Output tokens
The context window applies to both.
For example:
Context budget: 32K tokens
Conversation: 24K
Current request: 2K
Expected output: 4K
-------------------
Total: 30K
That may be acceptable.
But if the conversation grows to:
Conversation: 30K
Current request: 2K
Expected output: 4K
-------------------
Total: 36K
the request exceeds the intended budget.
A production application should therefore reserve space for output rather than consuming the entire context window with history.
Reduce the Maximum Conversation Context
A common mistake is assuming that a larger context window is always better.
It is not.
A large context window can increase:
- Memory requirements
- Processing time
- Cost
- Queueing
- Cache pressure
If users rarely need the full context window, allowing unlimited history simply increases resource consumption.
Define a practical context policy.
For example:
Maximum working context
|
+-- System instructions
+-- Conversation summary
+-- Recent messages
+-- Current request
+-- Output reservation
The exact limits depend on the model and workload.
Measure before choosing them.
Understand Prefix Caching
Many AI inference systems can reuse computation for requests that share the same prefix.
A prefix might contain:
System prompt
+
Common instructions
+
Stable context
If multiple requests use the same prefix, a serving engine may be able to reuse cached computation instead of processing the same prefix repeatedly.
This can be especially useful for applications where every request starts with a large system prompt.
For example:
Request A
System prompt
+
Product documentation
+
User question A
Request B
System prompt
+
Product documentation
+
User question B
The first part is identical.
If the inference engine supports prefix caching, that repeated prefix can become much cheaper to process.
However, prefix caching and KV-cache reuse are not exactly the same concept.
A prefix cache is about reusing computation for matching prefixes across requests.
A live KV cache also represents state associated with active inference sequences.
The distinction matters when planning memory and routing.
Avoid Changing Stable Prefixes Unnecessarily
Prefix caching works best when the beginning of requests remains stable.
Suppose the application generates:
System prompt
Current timestamp
Random request ID
User profile
Long documentation
User question
If values near the beginning change on every request, the reusable prefix becomes smaller.
A better structure may be:
Stable system instructions
Stable documentation context
Stable policy context
Dynamic user information
Current question
The stable content should appear before frequently changing content where the inference engine's caching strategy supports prefix reuse.
This is an application-level optimization that can improve inference efficiency without changing the model.
KV Cache Is Not the Same as Conversation Storage
Do not confuse the inference server's KV cache with your application's conversation database.
The application might store:
PostgreSQL
|
+-- Conversation ID
+-- Messages
+-- Summary
+-- User metadata
The inference server might maintain:
GPU memory
|
+-- Active KV cache
These have different lifecycles.
A conversation can exist in the database for months while its KV cache is discarded when the inference request ends.
This is important when designing recovery.
Do not assume that restarting an inference Pod preserves its KV cache.
If the Pod is replaced, the GPU memory containing the cache is lost.
The application must be able to reconstruct the required context from durable storage.
Why Kubernetes Routing Matters
Suppose your cluster has three inference Pods:
Pod A
Conversation X cache
Pod B
Conversation Y cache
Pod C
Conversation Z cache
A later request from Conversation X may benefit from being sent back to Pod A if the relevant cache is still available.
A normal load balancer may instead send it to Pod B.
That can eliminate the cache benefit.
This is one reason AI inference routing differs from ordinary web traffic.
The Kubernetes Gateway API Inference Extension was created specifically to address inference routing problems, including long-running sessions and in-memory token caches. Its design introduces inference-aware endpoint selection instead of relying only on conventional HTTP load balancing.
Session Affinity Can Help, But It Is Not a Complete Solution
You may be tempted to solve the problem with sticky sessions.
For example:
Conversation X -> Pod A
Conversation Y -> Pod B
Conversation Z -> Pod C
This can improve cache locality.
But it can also create uneven load.
Imagine:
Pod A
10 very long conversations
Pod B
2 short conversations
Pod C
1 short conversation
Simple affinity can leave Pod A overloaded.
A better inference-aware router can consider both locality and current workload.
The goal is not:
Always send the request to the same Pod.
The goal is:
Send the request to a Pod where the cached state is useful
without creating an unhealthy hotspot.
Queue Length Becomes Important
When KV-cache memory is heavily used, requests may begin waiting for available capacity.
For example:
GPU memory: 91%
Active requests: 8
Queue: 2
Then:
GPU memory: 94%
Active requests: 10
Queue: 8
And eventually:
GPU memory: 97%
Active requests: 12
Queue: 20
At this point, increasing request concurrency may make latency worse.
The system should instead consider:
- Reducing context
- Increasing GPU capacity
- Adding replicas
- Improving batching
- Using a more memory-efficient serving configuration
- Routing requests to less-loaded workers
Inference-aware routing systems can use live signals such as queue length and memory-related information when selecting endpoints.
Use Continuous Batching Carefully
Inference servers often batch multiple requests together to improve GPU utilization.
Traditional batch processing might look like:
Request A
Request B
Request C
Wait
Process A+B+C
Continuous batching is designed for inference workloads where requests arrive and finish at different times.
Conceptually:
Time 1
A B C
Time 2
A B C D
Time 3
A B D E
The server continuously adjusts which sequences are being processed.
This can increase throughput.
However, batching does not eliminate KV-cache memory requirements.
Each active sequence still consumes memory.
If you allow too many concurrent sequences, the cache can become the limiting resource.
Limit Concurrent Sequences
A practical inference server should have a controlled concurrency limit.
For example:
Maximum active sequences = 32
The exact value depends on:
- Model size
- GPU memory
- Context length
- Batch behavior
- Quantization
- KV-cache precision
- Expected output length
The important point is that unlimited concurrency is rarely a good production setting.
When the server reaches its safe capacity, additional requests should queue or be rejected according to the application's policy.
This is better than allowing memory pressure to turn into repeated out-of-memory failures.
Quantization Can Reduce Memory Pressure
Quantization reduces the numerical precision used by model weights and, depending on the inference stack, can also affect the memory required by other model state.
For example:
Higher precision
|
v
More memory
|
v
Fewer concurrent requests
With a suitable quantized model:
Lower memory requirement
|
v
More available GPU memory
|
v
Potentially more concurrency
However, quantization can affect output quality and performance.
Do not choose a quantization format only because it reduces memory usage.
Benchmark the actual model and workload.
KV Cache Quantization Is a Different Decision
Reducing model weight precision and reducing KV-cache precision are related but separate optimization decisions.
The model weights determine how the model itself is represented.
The KV cache stores intermediate attention state for active sequences.
Depending on the serving engine and model architecture, KV-cache quantization may reduce memory consumption while introducing its own quality and performance tradeoffs.
Do not assume that changing model precision automatically solves KV-cache pressure.
Measure the memory used by each component separately.
Use a Memory Budget
Instead of allowing the model server to consume almost all GPU memory, define a practical memory budget.
For example:
GPU memory
|
+-- Model weights
|
+-- Runtime memory
|
+-- KV cache
|
+-- Temporary workspace
|
+-- Safety margin
The exact percentages depend on the model server and hardware.
The important idea is to reserve memory for operations outside the steady-state KV cache.
Running at 99% memory utilization may look efficient until a slightly larger request arrives and causes an out-of-memory failure.
A small safety margin can improve reliability.
Watch for Fragmentation
GPU memory pressure is not always just about total allocated memory.
Inference engines manage memory dynamically as requests start and finish.
Long-running conversations with different context lengths can create complicated allocation patterns.
For example:
Request A -> 4K context
Request B -> 32K context
Request C -> 8K context
Request D -> 16K context
As requests finish and new requests arrive, the memory manager must reuse available space efficiently.
The serving engine's memory management strategy therefore matters.
Use the inference server's documented memory controls rather than assuming that a generic Kubernetes memory setting controls GPU allocation.
Kubernetes Memory Limits Do Not Control GPU Memory
This distinction causes confusion.
A Kubernetes manifest may contain:
resources:
limits:
memory: "32Gi"
nvidia.com/gpu: "1"
The memory limit refers to system RAM.
It does not mean:
GPU memory <= 32 GiB
GPU memory is managed separately by the accelerator runtime and model-serving process.
Therefore, monitoring should include both:
Node RAM
+
GPU memory
and the inference server's own memory metrics where available.
Scale Horizontally When One GPU Cannot Handle the Workload
If one GPU is consistently under KV-cache pressure, adding another inference replica may be more effective than allowing the first Pod to run at maximum memory.
For example:
Before
Traffic
|
v
GPU A
|
+-- 20 active conversations
After:
Traffic
|
v
Inference Router
|
+---- GPU A
|
+---- GPU B
Each worker handles fewer active conversations.
This can reduce queueing and cache pressure.
But horizontal scaling has a cost.
Each replica may need to load its own model weights.
If the model is large, adding replicas can consume substantial GPU memory and increase infrastructure cost.
Choose Between Larger GPUs and More GPUs
There are two basic strategies.
Use a larger GPU:
1 large GPU
|
+-- Larger model
+-- Larger KV cache
+-- More concurrency
Or use multiple GPUs:
GPU A
GPU B
GPU C
GPU D
The correct choice depends on the model architecture and serving engine.
A model that fits comfortably on one GPU may benefit from multiple replicas.
A model that cannot fit on one GPU may require tensor or pipeline parallelism.
Do not assume that horizontal replicas can replace a multi-GPU model configuration.
Handle Long-Running Conversations Explicitly
Long-running conversations should have a lifecycle.
For example:
New conversation
|
v
Active
|
v
Idle
|
v
Archived
Only active conversations need inference resources.
The database can retain the complete history while the inference service keeps only the state required for active requests.
This gives the application a useful separation:
Durable state
|
v
Database
Temporary inference state
|
v
GPU KV cache
When a conversation becomes idle, its active inference state can be discarded.
When the user returns, the application reconstructs the required context.
Use Conversation State Outside the Model Server
Do not make one model Pod the permanent owner of a conversation.
Suppose:
Conversation A -> Pod 1
Then Pod 1 fails.
If the application's only copy of the conversation state was inside that Pod, the conversation is lost.
Instead:
Conversation
|
+-- Database
|
+-- Summary
|
+-- Recent messages
|
v
Inference request
|
v
Any healthy model Pod
This makes the system resilient to Pod replacement.
The KV cache should be treated as an optimization, not the source of truth.
Use Cache-Aware Routing When It Provides a Real Benefit
Not every application needs sophisticated cache-aware routing.
If requests are short and conversations are small, ordinary load balancing may be sufficient.
Cache-aware routing becomes more useful when:
- Conversations are long.
- Requests are frequent.
- Prefixes are large.
- GPU memory is expensive.
- The same conversation generates many requests.
- Cache reuse significantly reduces latency.
The Kubernetes Gateway API Inference Extension roadmap explicitly includes prefix-cache-aware load balancing for remote caches, showing that cache locality is becoming an important consideration in inference networking.
This does not mean every Kubernetes cluster needs an inference gateway immediately.
Measure the workload first.
Monitor Cache Efficiency
A production inference system should track more than request latency.
Useful metrics include:
Active sequences
Queue length
Input tokens
Output tokens
Context length
GPU memory usage
KV-cache utilization
Cache hit rate
Prefix-cache reuse
Time to first token
Tokens per second
Request duration
Out-of-memory events
Pod restarts
The exact metrics depend on the model server.
The important part is connecting application behavior to infrastructure behavior.
For example:
Context length increased
|
v
KV cache increased
|
v
GPU memory increased
|
v
Concurrency decreased
|
v
Queue increased
|
v
Latency increased
That chain is much more useful than simply seeing that “GPU usage is high.”
Alert on Queue Growth Before Out-of-Memory
An out-of-memory event is already a failure.
A better monitoring strategy is to alert when the system approaches unsafe conditions.
For example:
KV cache utilization rising
+
Queue length rising
+
GPU memory near limit
This can trigger:
- Horizontal scaling
- Traffic shaping
- Lower concurrency
- Context reduction
- Load shedding
- Operator investigation
The exact thresholds should come from load testing.
Avoid copying arbitrary percentages into production.
Apply Backpressure
When the inference cluster reaches capacity, accepting unlimited requests is not helpful.
Without backpressure:
Traffic increases
|
v
Queue increases
|
v
Memory pressure
|
v
Latency increases
|
v
Timeouts
|
v
Retries
|
v
More traffic
This creates a feedback loop.
A better system applies backpressure:
Traffic increases
|
v
Capacity reached
|
+-- Queue within limit
|
+-- Reject lower-priority request
|
+-- Return retry response
|
+-- Route to another pool
Backpressure is particularly important for AI systems because one request can consume substantially more compute and memory than an ordinary HTTP request.
Do Not Let Clients Retry Aggressively
Suppose an inference request takes 15 seconds.
The client times out after 10 seconds and retries.
Now the server may be processing both:
Original request
+
Retry
If many clients do this, the workload can multiply rapidly.
Use sensible timeout and retry policies.
For interactive applications, the API layer should distinguish between:
Request still processing
and:
Request failed
Streaming responses can also help users receive output earlier instead of waiting for the entire response.
A Practical Architecture
A production-oriented design can look like this:
Client
|
v
API / AI Gateway
|
v
Inference Router
|
+----------+----------+
| |
v v
Model Pool A Model Pool B
| |
+----+----+ +----+----+
| | | |
v v v v
GPU Pod GPU Pod GPU Pod GPU Pod
| | | |
+----+----+-----------+---------+
|
v
Model Runtime
|
v
KV Cache
Outside the inference path:
Conversation Database
|
+-- Full history
+-- Summary
+-- Metadata
Monitoring:
Inference Metrics
|
v
Metrics / Observability
|
+-- GPU memory
+-- Queue
+-- KV cache
+-- Latency
+-- Tokens
This architecture makes an important distinction.
The database stores durable conversation state.
The inference server maintains temporary computation state.
The gateway handles traffic.
Kubernetes manages the infrastructure.
Each layer has a different responsibility.
Common Mistakes
Sending the complete conversation forever
Long histories consume more context and can increase inference memory and latency.
Treating the KV cache as durable storage
A Pod restart can remove the cache.
Store important conversation state outside the inference server.
Monitoring only GPU utilization
GPU memory and queue length can become bottlenecks before compute utilization reaches 100%.
Using unlimited concurrency
More concurrent sequences can eventually exhaust GPU memory.
Relying only on sticky sessions
Affinity can improve cache locality but may create hotspots.
Changing the system prompt on every request
Frequent changes near the beginning of the prompt can reduce prefix-cache reuse.
Using CPU memory limits as GPU protection
Kubernetes RAM limits do not control GPU memory.
Scaling without considering model weights
Every new replica may need its own copy of the model.
Retrying timed-out inference requests blindly
Retries can multiply expensive workloads.
Waiting for out-of-memory failures before scaling
Monitor memory and queue behavior before the server reaches a failure state.
Production Checklist
Before running long-context AI inference in production, verify:
- Conversation history is stored outside the inference Pod.
- The application has a clear context-window policy.
- Output token budget is reserved.
- Old messages can be summarized or removed.
- Context limits are based on measured workload behavior.
- Model and KV-cache memory are monitored separately.
- Active sequence limits are configured.
- Queue length is monitored.
- GPU memory pressure is monitored.
- Prefix caching is enabled where it provides measurable value.
- Stable prompt prefixes are kept consistent where practical.
- Cache-aware routing is considered for long-running conversations.
- Sticky routing does not create severe hotspots.
- Inference Pods can be replaced without losing durable conversation state.
- Autoscaling considers inference-specific metrics.
- Backpressure is implemented.
- Client retries are bounded.
- Streaming is used where appropriate for long responses.
- Model rollout procedures account for model-loading time.
- Load tests include long conversations and high concurrency.
- Out-of-memory behavior has been tested before production.
Summary
Long conversations create a different inference problem from ordinary API traffic.
As context grows, the inference server has more state to process and potentially more KV-cache memory to maintain. When many long-running conversations share the same GPU, that memory pressure can reduce concurrency, increase queueing, and eventually cause requests to fail.
The first optimization should usually happen before the request reaches the model.
Reduce unnecessary history.
Use summaries.
Reserve space for generated output.
Keep stable prefixes consistent when prefix caching can help.
Then optimize the inference layer.
Control concurrency.
Monitor KV-cache and GPU memory.
Use batching carefully.
Scale horizontally when additional GPU capacity is available.
Finally, optimize the Kubernetes networking layer.
Long-running inference requests and in-memory token caches make ordinary round-robin routing less suitable for some workloads. Kubernetes's Gateway API Inference Extension is designed around this problem and provides inference-specific routing concepts that can use signals such as queue length, memory usage, and cached model state when selecting inference endpoints.
The important architectural principle is simple:
Conversation database
|
v
Durable application state
Inference server
|
v
Temporary compute state
Kubernetes
|
v
Infrastructure management
Inference gateway
|
v
Traffic and routing
Do not try to solve KV-cache pressure with Kubernetes replicas alone.
The real solution is to control context size, manage concurrency, preserve useful cache locality, route requests intelligently, and scale GPU capacity based on the behavior of the actual inference workload.
Join the conversation! Your thoughts help the community grow.