Introduction
AI coding agents behave very differently from traditional chat-based assistants. A developer may provide one instruction, but the agent can perform dozens of model calls while inspecting files, searching a repository, editing code, running tests, reading failures, and deciding what to do next.
That creates a systems problem that is easy to overlook: how efficiently can the agent reuse the context it has already processed?
A recent production-scale study of GitHub Copilot coding-agent traces provides useful evidence. The study analyzed sampled traces from June 2026 covering about 13 million sessions, more than 3.2 million users, 761 million LLM calls, and 95 trillion tokens. It found that KV-cache reuse was substantially higher within an agent turn than across turn boundaries, while events such as model switching and context compaction could sharply reduce cache reuse.
For developers building AI agents, this matters because context caching is not merely an infrastructure optimization. Poor cache behavior can increase repeated computation, affect latency, and make long-running agent workloads more expensive to serve.
This article explains how context caching works at a high level, why compaction and model switching can disrupt it, and how developers can design agent workflows that make better use of stable context.
What Is Context Caching?
Large language models process a sequence of tokens to generate a response. During inference, systems can maintain intermediate state associated with previously processed tokens.
One important form of this state is the key-value (KV) cache used by transformer attention mechanisms.
A simplified request might look like this:
System instructions
+
Repository context
+
Conversation history
+
Tool result
|
v
Model
|
v
Response
If a subsequent model invocation contains a compatible prefix, the serving system may be able to reuse previously computed state instead of processing the entire prefix again.
Conceptually:
First request:
[A][B][C][D]
|
+--> Process and cache
Next request:
[A][B][C][E]
|
+--> Reuse A, B, C
+--> Process E
The exact caching mechanism depends on the serving infrastructure and model provider. The important application-level concept is prefix stability.
If the beginning of the request remains unchanged, there is more opportunity for reuse.
If the prefix changes substantially, previously computed state may no longer be useful.
Why Coding Agents Create a Unique Cache Workload
A normal chatbot often has a relatively simple interaction pattern:
User
|
v
LLM
|
v
Assistant
|
v
User
A coding agent behaves more like this:
User request
|
v
LLM
|
+--> Search files
|
v
LLM
|
+--> Read source
|
v
LLM
|
+--> Edit file
|
v
LLM
|
+--> Run tests
|
v
LLM
|
+--> Inspect failure
|
v
LLM
The production study found that coding-agent sessions frequently consist of autonomous LLM calls closely coupled with tool execution. It reported that about 87% of LLM calls in the analyzed workload were agent-initiated rather than directly initiated by the user.
That means one developer instruction can generate a substantial sequence of model invocations.
This makes cache reuse particularly important.
Cache Reuse Within an Agent Turn
The production study reported average KV-cache reuse of around 90% within a turn, while reuse fell to about 55% across turn boundaries.
The distinction is important.
Imagine an agent receiving:
Fix the failing OrderService tests.
It may then perform:
LLM 1 -> Inspect repository
LLM 2 -> Search OrderService
LLM 3 -> Read test file
LLM 4 -> Inspect implementation
LLM 5 -> Modify code
LLM 6 -> Run tests
LLM 7 -> Analyze failure
These calls are closely related and can share substantial context.
The next developer message may start a new turn:
Now also update the API validation.
The context structure may change enough that cache reuse becomes less effective.
This does not mean every new turn creates a complete cache miss. It means the workload boundary can reduce reuse.
Why Context Stability Matters
Consider two approaches.
Stable Context
System instructions
Repository rules
Project structure
Current task
Tool results
Latest change
The agent keeps the important prefix stable while adding new information.
Unstable Context
System instructions
Changing summary
Reordered repository information
New system instructions
Tool output
Rewritten history
Frequent changes to the beginning of the context can make reuse more difficult.
For agent developers, this suggests a practical rule:
Keep stable information stable whenever possible.
Repository instructions, tool definitions, and persistent agent configuration should not be regenerated or reordered unnecessarily between model calls.
Context Compaction
Long-running sessions eventually accumulate a large amount of information.
An agent may have processed:
20 source files
15 tool responses
8 test executions
5 build failures
4 implementation attempts
3 architectural decisions
Keeping every historical detail can become impractical.
Context compaction reduces the working history into a smaller representation.
For example:
Before compaction:
Task
|
+-- User request
+-- File inspections
+-- Tool calls
+-- Failed attempts
+-- Test output
+-- Design decisions
+-- Code changes
After compaction:
Task summary
|
+-- Objective
+-- Files changed
+-- Decisions
+-- Current failure
+-- Remaining work
Compaction can be useful, but it changes the context.
The production study found that context compaction occurred in a subset of sessions and was associated with cache cold-start behavior.
This creates an important trade-off:
Large context
|
+--> More historical information
|
+--> Potentially more reusable state
|
+--> Higher context-management burden
Compacted context
|
+--> Smaller working state
|
+--> Less historical detail
|
+--> Potential cache disruption
Compaction should therefore be treated as a systems event rather than merely a token-count optimization.
What Should Be Preserved During Compaction?
A good compacted state should preserve information that would be expensive for the agent to rediscover.
For example:
Task:
Add retry support to PaymentService.
Files changed:
PaymentService.cs
PaymentServiceTests.cs
Decision:
Use bounded exponential backoff.
Known issue:
Timeout test still fails.
Remaining work:
Fix timeout handling and rerun tests.
This is more useful than retaining hundreds of lines of compiler output.
A useful preservation strategy is:
| Context | Preserve | Reason |
|---|
| Original objective | Yes | Defines the task |
| Architectural decisions | Yes | Avoids repeated reasoning |
| Files modified | Yes | Maintains working state |
| Known failures | Yes | Prevents repeating failed approaches |
| Latest test status | Yes | Identifies remaining work |
| Raw logs | Usually no | Can often be regenerated |
| Duplicate tool output | No | Low information value |
| Temporary searches | Usually no | Often inexpensive to repeat |
The goal is not simply to make the context smaller.
The goal is to make the context information-dense.
Model Switching Can Invalidate Cache Reuse
Model switching is another important event.
Suppose an agent starts a task using Model A:
Task
|
v
Model A
|
+--> Tool calls
|
+--> More model calls
Later, the agent switches to Model B:
Task
|
v
Model B
Even if the textual context is identical, cache reuse is not guaranteed across different models.
The production study reported that model switches reduced cached context dramatically, with only about 8% remaining cached after a switch in the analyzed workload.
This has an important design implication.
Model switching should be treated as a potentially expensive event.
That does not mean developers should never switch models. It means switching should have a reason.
When Model Switching Makes Sense
There are legitimate reasons to change models.
For example:
Simple classification
|
v
Fast model
Complex debugging
|
v
More capable model
Or:
Primary model unavailable
|
v
Fallback model
The decision becomes a trade-off:
| Benefit | Cost |
|---|
| Better capability | Cache disruption |
| Lower per-request cost | Possible additional processing |
| Provider failover | Context reuse may decrease |
| Specialized model | Additional routing complexity |
A routing system should therefore consider not only model price and quality but also the state of the current agent session.
Designing Model Switching Carefully
A simple routing implementation might look like this:
public enum TaskComplexity
{
Simple,
Standard,
Complex
}
public sealed class ModelRouter
{
public string SelectModel(TaskComplexity complexity)
{
return complexity switch
{
TaskComplexity.Simple => "fast-model",
TaskComplexity.Standard => "general-model",
TaskComplexity.Complex => "reasoning-model",
_ => "general-model"
};
}
}
This works for independent requests.
For a long-running agent, however, you may want additional state:
public sealed record SessionState(
string CurrentModel,
int ToolCalls,
bool ContextCompacted);
The router can then consider whether a switch is worth the potential cache disruption.
public bool ShouldSwitch(
SessionState session,
TaskComplexity requestedComplexity)
{
if (session.ContextCompacted)
{
return true;
}
if (requestedComplexity == TaskComplexity.Complex &&
session.CurrentModel == "fast-model")
{
return true;
}
return false;
}
This is a policy example rather than a universal routing algorithm.
The correct decision depends on model capabilities, workload characteristics, and serving infrastructure.
Measuring Cache Efficiency
If you are building an agent platform, cache behavior should be measured directly where the serving stack exposes appropriate telemetry.
Useful metrics include:
| Metric | Purpose |
|---|
| Cache hit rate | Measures context reuse |
| Cache miss rate | Shows repeated processing |
| Prefix length | Shows how much context is reusable |
| Context size | Tracks workload growth |
| Compaction frequency | Identifies long-session pressure |
| Model switches | Identifies cache-disrupting events |
| Turn duration | Helps understand session behavior |
| Tool calls per turn | Shows agent activity |
A useful conceptual calculation is:
Cache Reuse %
=
Reusable Cached Context
-----------------------
Total Context Processed
× 100
The exact definition should match the telemetry exposed by the serving system.
Do not compare metrics from different platforms unless their cache accounting methods are equivalent.
Building a Context-Efficiency Benchmark
A controlled benchmark can compare different agent strategies.
For example:
Strategy A
No compaction
Strategy B
Periodic compaction
Strategy C
Event-driven compaction
Strategy D
Stable system context + selective compaction
Run the same task set against each strategy.
Record:
Task success rate
Total model calls
Context tokens
Cache hit rate
Compaction events
Model switches
Tool calls
Latency
Estimated compute
Then compare the complete workload.
A strategy that produces the smallest context is not automatically the best strategy.
For example:
Strategy A
Lower context
+
More repository rediscovery
=
More model/tool calls
while:
Strategy B
Larger retained state
+
Higher cache reuse
=
Fewer repeated operations
The correct choice depends on the workload.
A Practical Agent State Model
A useful design is to explicitly distinguish persistent state from transient context.
public sealed record AgentState(
string Task,
IReadOnlyList<string> ModifiedFiles,
IReadOnlyList<string> Decisions,
IReadOnlyList<string> KnownFailures,
string CurrentStep);
This state can be stored independently of the conversational history.
For example:
var state = new AgentState(
Task: "Fix payment retry behavior",
ModifiedFiles:
[
"PaymentService.cs",
"PaymentServiceTests.cs"
],
Decisions:
[
"Use bounded retries"
],
KnownFailures:
[
"Timeout test still fails"
],
CurrentStep: "Investigate timeout handling");
The advantage is that compaction does not have to reconstruct the entire project state from conversational history.
The application maintains an explicit representation of important state.
Reducing Unnecessary Cache Disruption
Several practical techniques can help.
Keep System Instructions Stable
Avoid rebuilding system-level instructions for every model call unless the content actually needs to change.
Keep Tool Definitions Stable
Tool schemas are part of the model's working context. Avoid unnecessary changes during a session.
Summarize Tool Results Selectively
Do not repeatedly inject huge logs when a concise structured result is sufficient.
Preserve Important Decisions
Architectural and implementation decisions should survive compaction.
Avoid Unnecessary Model Switching
Switch when capability, availability, or another strong requirement justifies it.
Separate Persistent State From Conversation History
Store important task state explicitly instead of relying entirely on the model's historical context.
Common Mistakes
Treating Context Compaction as Free
Compaction may reduce context size but can also change the context structure and reduce cache reuse.
Switching Models for Minor Reasons
A small quality or cost difference may not justify disrupting a long-running session.
Storing Everything in the Prompt
Large raw logs and duplicated tool responses increase context without necessarily improving decisions.
Relying Only on Conversation History
Important task state should be represented explicitly when the agent workflow is long-lived.
Measuring Only Token Count
Token reduction is useful, but task success, latency, tool activity, cache reuse, and rediscovery work also matter.
Best Practices
Keep stable context stable. Avoid unnecessary changes to system instructions, tool definitions, and persistent repository context.
Treat compaction as a workload event. Measure its effect instead of assuming it is free.
Preserve state, not raw history. Keep decisions, modified files, failures, and remaining work.
Avoid unnecessary model switches. Consider cache implications alongside model quality and price.
Separate persistent agent state from conversational history.
Filter large tool outputs. Give the model the information needed for its next decision.
Measure cache behavior. Use actual serving telemetry where available.
Benchmark complete agent sessions. Compare task completion, latency, context processing, and tool activity together.
Use event-driven compaction carefully. Compact when the retained context stops providing enough value, not simply on an arbitrary schedule.
Design for rediscovery cost. Information that is expensive to reconstruct deserves stronger persistence.
Frequently Asked Questions
Does a larger context always mean worse cache performance?
No. A larger context can contain substantial reusable information. The important factors include prefix stability, cache capacity, context changes, and the serving system's caching strategy.
Does every model switch destroy the entire cache?
Not necessarily in every implementation. Cache behavior is infrastructure-dependent. However, the production GitHub Copilot study found model switching to be strongly associated with cache invalidation in its analyzed workload.
Should AI agents avoid context compaction?
No. Long-running agents need mechanisms to control context growth. The better approach is to compact intelligently while preserving important state.
Is KV-cache optimization something application developers need to care about?
Developers building or operating agent infrastructure should care about it because application behavior can influence context stability, model switching, compaction, and tool-call patterns. Developers simply consuming a hosted coding assistant generally have much less direct control over the underlying serving cache.
What is the biggest lesson from production coding-agent workloads?
Agentic workloads should not be treated like ordinary chat traffic. Production evidence shows that autonomous tool loops, session boundaries, model switches, context compaction, and long-tailed workloads create distinct infrastructure requirements.
Conclusion
AI coding agents introduce a different kind of workload from traditional chat applications. A single developer request can trigger a long sequence of model and tool calls, making context reuse an important part of system efficiency.
Production-scale GitHub Copilot traces show that cache reuse can be high within an agent turn but significantly lower across turn boundaries, while model switches and context compaction can create substantial cache disruption.
For developers designing agent systems, the practical response is not to eliminate compaction or model switching. Both can be valuable. Instead, treat them as deliberate architectural decisions.
Keep stable context stable, preserve important state explicitly, avoid unnecessary model changes, control large tool outputs, and measure the complete workload. When cache behavior is considered alongside latency, tool calls, context size, and task success, AI coding agents become much easier to reason about and optimize.