AI model selection is no longer just a configuration decision. In a production application, the model you choose can affect response quality, latency, throughput, and operating cost. Sending every request to the most capable model may be simple, but it is not always the best architecture.
A model-routing strategy allows an application to select an appropriate model based on the request. Simple tasks can use a lower-cost model, while complex requests can be sent to a more capable model. Microsoft.Extensions.AI provides the IChatClient abstraction and composable chat-client pipeline that make this kind of architecture easier to implement in .NET.
This article shows how to build a practical benchmark for AI model routing in .NET and measure three important dimensions: cost, latency, and response quality.
Research gap: Microsoft.Extensions.AI provides the underlying IChatClient abstraction and pipeline infrastructure, but the exact routing implementation available to an application can vary by package/version. Therefore, the benchmark below uses an explicit routing layer rather than assuming a particular preview-only routing API. This keeps the benchmark reproducible and avoids attributing unsupported behavior to the framework.
Why Benchmark AI Model Routing?
A routing strategy sounds straightforward:
Receive a request.
Decide which model should handle it.
Send the request to that model.
Return the response.
The difficult part is deciding whether the routing strategy actually improves the application.
Suppose an application has two models:
| Model | Relative Cost | Expected Capability | Typical Use |
|---|
| Model A | Lower | Good | Simple questions, classification |
| Model B | Higher | Better | Complex reasoning, coding |
| Static Model A | Lower | Fixed | Every request |
| Static Model B | Higher | Fixed | Every request |
| Router | Variable | Adaptive | Selects based on request |
A router is useful only if its decision quality justifies the additional complexity.
Microsoft's model-routing guidance describes cost-optimized, quality-optimized, and balanced routing as common strategies.
That makes benchmarking important. Instead of asking, "Which model is best?", a better engineering question is:
Which routing strategy provides the required quality at an acceptable latency and cost?
Understanding Microsoft.Extensions.AI
Microsoft.Extensions.AI provides common abstractions for working with AI services in .NET. One of the most important abstractions for chat applications is IChatClient.
The interface represents a chat client and supports operations such as regular and streaming responses.
A provider-specific implementation can be converted into an IChatClient. For example, an OpenAI-compatible chat client can be adapted using AsIChatClient().
This gives the application an architecture like this:
Application
|
v
IChatClient
|
+--------------------+
| |
v v
Model A Model B
| |
Provider A Provider B
The application does not need to know every provider-specific API.
This abstraction is particularly useful for benchmarking because the application code can remain mostly unchanged while the underlying model changes.
Designing the Benchmark
A useful benchmark should compare the same workload across multiple routing strategies.
For example:
Test Dataset
|
+----> Static Model A
|
+----> Static Model B
|
+----> Rule-Based Router
|
+----> Quality-Based Router
Every strategy should receive the same test prompts.
The benchmark should record at least:
For latency, measure at least p50 and p95 rather than only the average. Average latency can hide slow outliers that are important in production.
Creating the .NET Project
Create a console application:
dotnet new console -n AiRoutingBenchmark
cd AiRoutingBenchmark
Install the required AI abstraction and provider packages appropriate for the models you want to test.
The exact package versions should be pinned in the project because Microsoft.Extensions.AI packages can expose preview APIs that change over time. The current API documentation itself marks some package versions as prerelease, so production benchmarks should record the exact package versions used.
A simplified project might look like this:
<ItemGroup>
<PackageReference Include="Microsoft.Extensions.AI" Version="10.7.0" />
<PackageReference Include="Microsoft.Extensions.AI.OpenAI" Version="10.7.0" />
</ItemGroup>
Use the package versions that are actually available and supported in your target environment rather than blindly copying a version from an article.
Creating Multiple IChatClient Instances
The main advantage of the abstraction becomes clear when registering multiple clients.
using Microsoft.Extensions.AI;
using OpenAI.Chat;
IChatClient fastModel =
new ChatClient(
"model-a",
Environment.GetEnvironmentVariable("MODEL_A_API_KEY"))
.AsIChatClient();
IChatClient qualityModel =
new ChatClient(
"model-b",
Environment.GetEnvironmentVariable("MODEL_B_API_KEY"))
.AsIChatClient();
The application now has two clients that expose the same interface.
That means the routing logic does not need to understand provider-specific request classes.
Building a Simple Model Router
A first benchmark should use a deterministic router.
For example, requests containing simple informational questions can use the lower-cost model, while coding or reasoning requests can use the more capable model.
public sealed class ModelRouter
{
private readonly IChatClient _fastClient;
private readonly IChatClient _qualityClient;
public ModelRouter(
IChatClient fastClient,
IChatClient qualityClient)
{
_fastClient = fastClient;
_qualityClient = qualityClient;
}
public IChatClient SelectModel(string prompt)
{
if (prompt.Contains("code", StringComparison.OrdinalIgnoreCase) ||
prompt.Contains("debug", StringComparison.OrdinalIgnoreCase) ||
prompt.Contains("architecture", StringComparison.OrdinalIgnoreCase))
{
return _qualityClient;
}
return _fastClient;
}
}
This is intentionally simple.
The objective at this stage is not to build the smartest router. It is to establish a measurable baseline.
Once the baseline is available, more sophisticated routing policies can be evaluated.
Measuring Latency
Use Stopwatch to measure end-to-end request latency.
using System.Diagnostics;
var stopwatch = Stopwatch.StartNew();
var client = router.SelectModel(prompt);
var response = await client.GetResponseAsync(prompt);
stopwatch.Stop();
Console.WriteLine(
$"Model: {client.GetType().Name}, " +
$"Latency: {stopwatch.ElapsedMilliseconds} ms");
For a meaningful benchmark, do not run a single request and treat the result as representative.
Run the same prompt set multiple times and record every result.
A benchmark record can be represented as:
public sealed record BenchmarkResult(
string PromptId,
string Model,
long LatencyMs,
int InputTokens,
int OutputTokens,
decimal EstimatedCost,
double QualityScore);
This makes the raw benchmark data easier to export to CSV, a database, or an analytics dashboard.
Measuring Cost
Cost should be calculated from the actual token usage and the pricing assumptions applicable to the models being tested.
A simplified calculation is:
decimal CalculateCost(
int inputTokens,
int outputTokens,
decimal inputPrice,
decimal outputPrice)
{
return
(inputTokens / 1_000_000m) * inputPrice +
(outputTokens / 1_000_000m) * outputPrice;
}
Do not hard-code pricing into application logic in production.
Instead, maintain model pricing as configuration:
{
"Models": {
"ModelA": {
"InputPricePerMillion": 0.0,
"OutputPricePerMillion": 0.0
},
"ModelB": {
"InputPricePerMillion": 0.0,
"OutputPricePerMillion": 0.0
}
}
}
This also makes historical benchmark results easier to reproduce because the pricing assumptions can be stored alongside the test run.
Measuring Response Quality
Cost and latency are not enough.
A router that reduces cost by sending almost everything to a weak model may look excellent financially while making the application worse.
Create a fixed evaluation dataset containing representative requests:
Q001 - Simple factual question
Q002 - Summarization
Q003 - C# coding task
Q004 - Debugging problem
Q005 - Architecture question
Q006 - Structured data extraction
Q007 - Multi-step reasoning
Then score each response using a consistent evaluation method.
For example:
| Metric | Score |
|---|
| Correctness | 0–5 |
| Relevance | 0–5 |
| Completeness | 0–5 |
| Instruction following | 0–5 |
The important point is consistency. The same evaluation criteria should be applied to every routing strategy.
Benchmarking Static Selection vs Routing
Now compare three strategies:
| Strategy | Model Selection | Cost | Latency | Quality |
|---|
| Static Cheap | Always Model A | Measure | Measure | Measure |
| Static Quality | Always Model B | Measure | Measure | Measure |
| Dynamic Router | Select per request | Measure | Measure | Measure |
The results should be evaluated as a trade-off rather than a single winner.
For example, a router might reduce the number of requests sent to the expensive model while maintaining similar quality. That would be a meaningful result even if the router itself introduces some decision overhead.
Do not publish fabricated percentage improvements. Run the benchmark against your actual workload and report the observed values.
Improving the Router
After establishing the baseline, routing can become more sophisticated.
Rule-Based Routing
Rules can classify requests using known characteristics:
Simple FAQ -> Model A
Classification -> Model A
Summarization -> Model A/B
Complex coding -> Model B
High-risk request -> Model B
This approach is predictable and easy to debug.
Classifier-Based Routing
A lightweight classifier can estimate request complexity before selecting the final model.
User Request
|
v
Complexity Classifier
|
+---- Simple ------> Model A
|
+---- Complex -----> Model B
The classifier itself introduces latency and cost, so it must be included in the benchmark.
Quality-Aware Routing
A more advanced design can evaluate the first response and retry with a stronger model when quality signals are poor.
Request
|
Model A
|
Quality Check
|
+---- Pass ----> Response
|
+---- Fail ----> Model B
This can be useful for workloads where most requests are simple but occasional requests require stronger reasoning.
Adding ChatClient Pipeline Instrumentation
Microsoft.Extensions.AI supports composable chat-client pipelines through ChatClientBuilder. The builder can construct a pipeline where calls pass through multiple stages.
This is useful for adding logging, telemetry, caching, or other cross-cutting behavior around AI calls.
For example:
builder.Services
.AddChatClient(fastModel)
.UseLogging();
The exact pipeline configuration depends on the packages and extensions used by the application.
The key architectural idea is to keep routing and observability separate. The router decides where the request goes; instrumentation records what happened.
Production Benchmark Considerations
A local benchmark is useful for development, but production routing requires more discipline.
Track:
A particularly useful metric is cost per successful high-quality response.
This is more meaningful than simply measuring cost per API call because a cheap response that fails evaluation may create another request and increase the real cost.
Advantages
Reduces unnecessary use of expensive models.
Provides a single abstraction around different AI clients.
Allows model selection to evolve independently from business logic.
Makes cost, latency, and quality measurable.
Supports gradual experimentation with different routing policies.
Can provide a foundation for provider fallback and resilience.
Disadvantages
Routing adds architectural complexity.
A classifier or evaluation step can introduce additional latency and cost.
Poor routing rules can reduce response quality.
Model capabilities and pricing change over time.
Benchmark results can become stale as models are updated.
Quality evaluation is harder than measuring latency or token usage.
Common Mistakes
Benchmarking Different Workloads
Comparing one model on simple prompts against another model on complex prompts produces misleading results.
Use the same evaluation dataset for every strategy.
Measuring Only Average Latency
Average latency can hide slow requests.
Always capture a distribution and report at least p50 and p95 for meaningful production analysis.
Ignoring Router Overhead
If the routing classifier takes 300 milliseconds, that time belongs in the end-to-end latency calculation.
Optimizing Only for Cost
The cheapest model is not automatically the best model.
Quality must remain a first-class metric.
Hard-Coding Model Pricing
Pricing changes. Keep pricing assumptions configurable and record them with benchmark results.
Troubleshooting
Results Change Between Benchmark Runs
LLM responses are not necessarily deterministic, and provider-side infrastructure can also vary.
Use a sufficiently large test set, control generation parameters where appropriate, and repeat runs before drawing conclusions.
The Router Always Selects One Model
Log the routing decision for every request.
A routing distribution such as:
Model A: 92%
Model B: 8%
can reveal whether the router is actually differentiating workloads.
Cost Results Do Not Match Provider Billing
Check whether the benchmark uses the same token accounting and pricing assumptions as the provider's billing system.
Treat benchmark cost as an estimate unless it is reconciled with authoritative billing data.
Best Practices
Start with a deterministic baseline router.
Use a fixed and representative evaluation dataset.
Measure cost, latency, quality, and errors together.
Capture p50 and p95 latency.
Record the selected model for every request.
Include routing overhead in end-to-end measurements.
Store package versions and benchmark configuration.
Re-run benchmarks when models, prompts, or routing policies change.
Keep pricing configuration separate from routing code.
Add automated quality evaluation before optimizing aggressively for cost.
Frequently Asked Questions
What is AI model routing?
AI model routing is the process of dynamically selecting an AI model for a request based on characteristics such as complexity, cost, latency requirements, or required quality.
Why use Microsoft.Extensions.AI for model routing?
Microsoft.Extensions.AI provides the IChatClient abstraction, allowing application code to work against a common chat-client interface instead of tightly coupling business logic to one provider.
Should every request use the most powerful model?
No. If a lower-cost model can reliably handle a workload, routing simple requests to that model can improve the application's cost profile.
What should be measured in a routing benchmark?
At minimum, measure latency, token usage, estimated cost, success rate, and response quality. Model-selection distribution is also valuable for understanding whether the router behaves as expected.
Is rule-based routing enough for production?
It can be, particularly when workload categories are well understood. More advanced classifiers or quality-aware routing should be introduced only when they provide measurable value.
Conclusion
AI model routing should be treated as an engineering optimization problem rather than simply a provider-selection feature. The important question is not whether one model is better than another, but whether the application can select the right model for each workload while maintaining acceptable quality, latency, and cost.
Microsoft.Extensions.AI provides a useful foundation through IChatClient and its composable client pipeline. A practical benchmark can then build on that abstraction to compare static model selection against deterministic or more advanced routing strategies.
The most important step is to measure before optimizing. A routing strategy should earn its place in the architecture by demonstrating an observable improvement in the metrics that matter to the application.