Modern .NET applications increasingly use more than one AI model.
A single application may have access to:
A high-capability model for complex reasoning
A faster model for simple requests
A lower-cost model for routine workloads
A specialized model for a particular task
A fallback model for availability problems
The engineering challenge is deciding which model should handle each request.
A static model-selection strategy is straightforward:
Application
|
v
Selected Model
|
v
Response
A routing strategy adds another decision layer:
Application
|
v
Routing Layer
|
+---- Model A
+---- Model B
+---- Model C
|
v
Response
The advantage is flexibility, but routing also introduces additional decision logic and potentially additional latency.
Microsoft.Extensions.AI provides the IChatClient abstraction and composable chat-client pipelines, which makes it possible to place routing, logging, retry, configuration, and other behaviors around AI clients. The current API also provides ChatClientBuilder and DelegatingChatClient as mechanisms for composing these pipelines.
This article explains how to benchmark a routing-based chat client against static model selection and determine whether routing actually improves the application's overall performance and economics.
Introduction
Suppose an application supports three request categories:
Simple Question
|
v
Fast Model
Complex Reasoning
|
v
Advanced Model
Fallback
|
v
Backup Model
A static strategy might send every request to the same model:
Every Request
|
v
Model A
A routing strategy might inspect the request and select a model:
+--> Model A
|
Request --> Router --+--> Model B
|
+--> Model C
The routing strategy can potentially improve cost, latency, or availability.
However, the router itself has a cost.
It may introduce:
Classification latency
Additional model calls
More complex configuration
More difficult debugging
Different behavior across workloads
Therefore, routing should be treated as an engineering hypothesis that needs to be benchmarked.
What Is Static Model Selection?
Static model selection means the application chooses the model before processing the request and does not dynamically change that choice.
For example:
IChatClient client = primaryClient;
var response = await client.GetResponseAsync(
"Explain dependency injection in .NET.",
cancellationToken: cancellationToken);
The application knows exactly which client will process the request.
This approach is simple and predictable.
The execution path is:
Request
|
v
Static Selection
|
v
Model
|
v
Response
What Is Routing?
Routing introduces a decision mechanism between the application and the underlying model clients.
Request
|
v
Router
|
+--> Fast Model
|
+--> Capable Model
|
+--> Fallback Model
The router can use different strategies.
For example:
Request Complexity
|
+--> Simple ----> Fast Model
|
+--> Moderate -> Balanced Model
|
+--> Complex --> Advanced Model
Another strategy could use availability:
Primary Model
|
X
Unavailable
|
v
Fallback Model
A third strategy could combine both:
Complexity
+
Cost
+
Availability
+
Latency
|
v
Model Selection
Microsoft.Extensions.AI and IChatClient
The IChatClient abstraction provides a common interface for chat model interactions. This allows application code to work against an abstraction instead of depending directly on a specific model implementation.
This is important for benchmarking because the application can execute the same workload against different client configurations.
For example:
public interface IModelExecutor
{
Task<ChatResponse> ExecuteAsync(
string prompt,
CancellationToken cancellationToken);
}
A static implementation can wrap one model.
A routing implementation can select among several models.
The benchmark can then compare both using the same test scenarios.
Static Selection Architecture
A simple static architecture looks like this:
Application
|
v
IChatClient
|
v
Model A
The advantage is that there is almost no selection overhead.
The main limitation is that every request follows the same model path unless application code explicitly changes the client.
Routing Architecture
A routing architecture looks like this:
Application
|
v
Routing Layer
|
+---------+---------+
| | |
v v v
Model A Model B Model C
The router becomes responsible for determining the target.
This can be implemented as a custom IChatClient wrapper or as a component in a chat-client pipeline.
DelegatingChatClient is specifically designed as a base type for clients that wrap another IChatClient, and the chat-client pipeline can be composed using ChatClientBuilder.Use(...).
Define the Benchmark Question
Before measuring anything, define what the benchmark is trying to prove.
For example:
Does dynamic routing reduce cost without causing unacceptable latency or quality degradation compared with static model selection?
That question produces several measurable dimensions:
Latency
Cost
Quality
Success Rate
Fallback Rate
Routing Accuracy
Throughput
Without a clear hypothesis, it is easy to produce a benchmark that generates numbers without providing an engineering conclusion.
Benchmark Scenarios
Use multiple workload categories.
Simple Requests
Examples:
What is dependency injection?
Convert this JSON into a C# record.
What does HTTP 404 mean?
Complex Requests
Examples:
Analyze this architecture and identify scalability risks.
Explain the tradeoffs between two distributed-system designs.
Review this code and identify concurrency problems.
Long-Context Requests
These contain larger amounts of input and can expose different model behavior.
Failure Scenarios
Simulate:
Timeout
Rate Limit
Unavailable Model
Invalid Response
Transient Network Failure
Routing should be evaluated not only when everything works but also when the preferred model fails.
Establish a Baseline
The first benchmark should use static selection.
For example:
All Requests
|
v
Model A
Measure:
p50 latency
p95 latency
p99 latency
Token usage
Cost
Success rate
Quality score
This becomes the baseline.
Then execute the same workload through the router.
All Requests
|
v
Router
|
+--> Model A
+--> Model B
+--> Model C
The two measurements can then be compared.
Benchmark Harness
Create a common interface.
public interface IBenchmarkClient
{
string Name { get; }
Task<BenchmarkResponse> ExecuteAsync(
BenchmarkRequest request,
CancellationToken cancellationToken);
}
The request can contain:
public sealed record BenchmarkRequest(
string Id,
string Category,
string Prompt);
The response can contain:
public sealed record BenchmarkResponse(
string RequestId,
string Model,
TimeSpan Latency,
long InputTokens,
long OutputTokens,
bool Success);
This provides a consistent measurement format.
Measure Latency
Use a monotonic timer.
var start = Stopwatch.GetTimestamp();
var response = await client.ExecuteAsync(
request,
cancellationToken);
var elapsed =
Stopwatch.GetElapsedTime(start);
This measures application-observed execution time.
Do not include unrelated operations such as loading the benchmark dataset or writing the final report inside the timed region.
Measure Routing Overhead Separately
Routing latency should not be hidden.
Consider:
Total Routed Latency
=
Routing Decision
+
Model Request
+
Response Processing
If routing itself requires another model call:
Total Latency
=
Router Model Call
+
Target Model Call
That additional call can be significant.
If routing is rule-based:
Total Latency
=
Rule Evaluation
+
Target Model Call
The difference can be substantial.
Therefore, capture routing time independently:
var routingStart = Stopwatch.GetTimestamp();
var target = await router.SelectAsync(
request,
cancellationToken);
var routingLatency =
Stopwatch.GetElapsedTime(routingStart);
Then measure the actual model request separately.
Benchmark Static Selection
A static benchmark might look like:
public async Task<BenchmarkResponse> RunStaticAsync(
BenchmarkRequest request,
IChatClient client,
CancellationToken cancellationToken)
{
var start = Stopwatch.GetTimestamp();
var response = await client.GetResponseAsync(
request.Prompt,
cancellationToken: cancellationToken);
var latency =
Stopwatch.GetElapsedTime(start);
return new BenchmarkResponse(
request.Id,
"static-model",
latency,
GetInputTokens(response),
GetOutputTokens(response),
true);
}
The exact token-usage extraction depends on the provider and client implementation.
The benchmark should use the actual usage metadata available from the selected client rather than estimating token counts from string length.
Benchmark Routing
A routing benchmark follows the same measurement boundary:
public async Task<BenchmarkResponse> RunRoutedAsync(
BenchmarkRequest request,
IRoutingClient client,
CancellationToken cancellationToken)
{
var start = Stopwatch.GetTimestamp();
var response = await client.GetResponseAsync(
request.Prompt,
cancellationToken);
var latency =
Stopwatch.GetElapsedTime(start);
return new BenchmarkResponse(
request.Id,
response.Model,
latency,
response.InputTokens,
response.OutputTokens,
true);
}
The important point is that both strategies receive the same benchmark request.
Rule-Based Routing
The simplest routing approach uses deterministic rules.
public string SelectModel(BenchmarkRequest request)
{
return request.Category switch
{
"Simple" => "fast",
"Complex" => "advanced",
"LongContext" => "long-context",
_ => "fast"
};
}
This has almost no classification overhead.
It is also easy to test.
The disadvantage is that rules can become increasingly complicated as workloads grow.
LLM-Based Routing
A more dynamic strategy can use a model to classify the request.
User Request
|
v
Routing Model
|
+--> Simple
+--> Complex
+--> Specialized
|
v
Target Model
This can be flexible but introduces an additional model operation.
For example:
Routing Decision = 80 ms
Target Model = 600 ms
Total = 680 ms
If static selection requires only:
Target Model = 600 ms
the router has made the request slower.
Routing must therefore generate enough savings elsewhere to justify its own overhead.
Benchmark Routing Accuracy
A routing system should also be evaluated for decision quality.
Create an expected model category for each benchmark request:
public sealed record RoutingExpectation(
string RequestId,
string ExpectedRoute);
Then compare:
Expected Route
vs
Selected Route
Calculate:
Routing Accuracy =
Correct Decisions
-----------------
Total Decisions
A router that selects the wrong model frequently may not produce the expected cost or quality benefits.
Measure Model Quality
Latency alone is not sufficient.
Suppose:
Static Model
Latency: 800 ms
Quality: 0.95
Routing
Latency: 650 ms
Quality: 0.86
Routing is faster, but the quality regression may be unacceptable.
Depending on the application, measure:
Task success rate
Structured-output validity
Groundedness
Answer relevance
Domain-specific correctness
Human evaluation
Retrieval quality for RAG workloads
The exact evaluation metric should match the application.
Cost Measurement
For each model request, capture:
Input Tokens
Output Tokens
Model
Pricing Version
Calculated Cost
Then aggregate by route.
Static Strategy
--------------
Total Cost
Average Cost
Cost Per Successful Task
Routing Strategy
----------------
Total Cost
Average Cost
Cost Per Successful Task
The most useful comparison is often:
Cost Per Successful Task
rather than simply cost per API request.
Example Cost Comparison
Imagine a benchmark produces:

Join the conversation! Your thoughts help the community grow.