Introduction
Modern .NET applications increasingly use multiple AI models instead of sending every request to the same model.
A simple application might always use one model:
User Request
|
v
Single AI Model
|
v
Response
That approach is easy to implement, but it may not be the best choice when workloads contain very different types of requests.
A production application could receive:
Simple classification requests
Short summarization tasks
Code-generation requests
Long-context analysis
Complex reasoning problems
High-volume, latency-sensitive requests
Using one model for everything can increase cost or latency. This is where semantic routing becomes interesting.
Instead of statically selecting a model, the application first evaluates the request and dynamically chooses the model that best fits the workload.
+--> Fast Model
|
User Request --> Router -+--> Balanced Model
|
+--> Reasoning Model
But semantic routing also introduces additional processing, latency, complexity, and failure modes.
The important question is therefore not simply:
Is semantic routing better than static model selection?
A more useful engineering question is:
Under which workloads does semantic routing provide enough quality or cost benefit to justify its routing overhead?
This article explains how to benchmark both approaches in a .NET application and how to evaluate cost, latency, quality, routing accuracy, and failure behavior.
Static Model Selection vs Semantic Routing
Static Model Selection
With static selection, the application chooses a model based on application configuration or a fixed business rule.
public async Task<string> GenerateAsync(string prompt)
{
return await _aiClient.GenerateAsync(
model: "balanced-model",
prompt);
}
Every request follows the same path:
Request
|
v
Configured Model
|
v
Response
The primary advantages are simplicity and predictability.
There is no routing decision to calculate, no additional classifier, and generally no additional network request.
Semantic Routing
Semantic routing evaluates the meaning or characteristics of the request before selecting a model.
Request
|
v
Semantic Router
|
+---- Simple ------> Fast Model
|
+---- Normal ------> Balanced Model
|
+---- Complex -----> Reasoning Model
A conceptual implementation could look like:
public async Task<string> GenerateAsync(string prompt)
{
var route = await _router.SelectModelAsync(prompt);
return await _aiClient.GenerateAsync(
model: route.Model,
prompt);
}
The router could use embeddings, a classifier, an LLM-based classifier, heuristics, or a combination of these techniques.
Why Benchmarking Matters
It is easy to assume that routing saves money because simple requests can use cheaper models.
However, routing itself has a cost.
For example:
Static Selection
Request
|
v
Model
|
v
Response
Semantic Routing
Request
|
v
Router
|
v
Selected Model
|
v
Response
The second architecture adds another stage.
That stage can introduce:
Additional latency
Additional token usage
Additional API calls
Routing errors
Operational complexity
New failure modes
A benchmark should therefore measure the entire request lifecycle rather than comparing only the selected model's response time.
Defining the Benchmark
A useful benchmark should contain representative workload categories.
For example:
| Workload | Complexity | Expected Model |
|---|
| Classification | Low | Fast |
| Short summarization | Low | Fast |
| FAQ response | Low | Fast |
| Code explanation | Medium | Balanced |
| Refactoring | Medium | Balanced |
| Architecture analysis | High | Reasoning |
| Complex debugging | High | Reasoning |
| Large document analysis | High | Reasoning |
The exact categories should come from real application traffic whenever possible.
Synthetic prompts can be useful for controlled experiments, but production-like workloads generally provide better routing insights.
Establishing the Static Baseline
First measure the application without semantic routing.
For example:
var stopwatch = Stopwatch.StartNew();
var response = await client.GenerateAsync(
model: "balanced-model",
prompt);
stopwatch.Stop();
Console.WriteLine(
$"Latency: {stopwatch.ElapsedMilliseconds} ms");
The baseline should capture at least:
This gives you a reference point.
Without a baseline, a routing experiment cannot demonstrate whether the additional complexity actually improves the system.
Measuring Semantic Routing Overhead
The router should be measured separately.
var routingTimer = Stopwatch.StartNew();
var route = await router.SelectModelAsync(prompt);
routingTimer.Stop();
var modelTimer = Stopwatch.StartNew();
var response = await client.GenerateAsync(
model: route.Model,
prompt);
modelTimer.Stop();
You can then calculate:
Total Latency
=
Routing Latency
+
Model Latency
+
Other Processing
This distinction is important.
Suppose semantic routing reduces model latency by 400 ms but adds 250 ms of routing overhead.
The application-level improvement is only approximately 150 ms before considering other effects.
Optimizing the selected model alone would therefore produce a misleading benchmark.
Measuring Routing Accuracy
Routing accuracy is one of the most important metrics.
Suppose your benchmark contains 1,000 requests and domain experts classify them into expected model categories.
You can compare:
Expected Route
vs
Actual Route
For example:
| Request | Expected | Actual | Correct |
|---|
| Short summary | Fast | Fast | Yes |
| Code review | Balanced | Fast | No |
| Architecture analysis | Reasoning | Reasoning | Yes |
| Simple classification | Fast | Balanced | No |
A basic routing accuracy metric is:
Routing Accuracy =
Correct Routes / Total Requests
But raw accuracy is not always enough.
A router that is wrong on difficult reasoning tasks can be much more harmful than one that occasionally sends a simple request to a slightly more expensive model.
Measuring Over-Routing
An important metric is over-routing.
Over-routing occurs when the router selects a more expensive or more powerful model than necessary.
For example:
Simple Request
|
v
Reasoning Model
The request may succeed, but the application has unnecessarily consumed additional resources.
Measure:
Over-Routing Rate =
Unnecessarily Expensive Routes / Total Requests
This is particularly important when the goal of routing is cost optimization.
Measuring Under-Routing
Under-routing is potentially more serious.
It happens when a complex request is sent to a model that cannot reliably handle the workload.
Complex Debugging Request
|
v
Fast Model
The response may be faster and cheaper, but the quality can deteriorate.
Measure:
Under-Routing Rate =
Insufficient Model Selections / Total Requests
For production systems, the benchmark should treat quality failures as first-class metrics rather than considering every successful HTTP response a success.
Cost Benchmark
A routing system should calculate the cost of the complete request.
For static selection:
Static Cost
=
Model Input Cost
+
Model Output Cost
For semantic routing:
Routing Cost
+
Selected Model Cost
Therefore:
Total Routing Cost
=
Router Input/Output Cost
+
Selected Model Input/Output Cost
A simple benchmark record might contain:
public sealed record BenchmarkResult(
string Strategy,
string Workload,
string Model,
double RoutingLatencyMs,
double ModelLatencyMs,
int InputTokens,
int OutputTokens,
double EstimatedCost,
bool Success);
This makes the results easier to aggregate.
Measuring Quality
Cost and latency alone are insufficient.
A routing system could reduce cost by always selecting the cheapest model, but that would not make it a successful optimization.
Quality can be evaluated using:
For coding workloads, a particularly useful technique is to evaluate the generated code against automated tests.
For example:
Prompt
|
v
Selected Model
|
v
Generated Code
|
v
Build
|
v
Unit Tests
This is generally more useful than judging code quality only from the generated text.
A Practical Benchmark Harness
A simple .NET benchmark model could look like this:
public sealed class BenchmarkRunner
{
public async Task<BenchmarkResult> RunAsync(
string strategy,
string prompt)
{
var routingLatency = 0.0;
var selectedModel = "balanced-model";
if (strategy == "semantic")
{
var routingTimer = Stopwatch.StartNew();
selectedModel =
await _router.SelectModelAsync(prompt);
routingTimer.Stop();
routingLatency =
routingTimer.Elapsed.TotalMilliseconds;
}
var modelTimer = Stopwatch.StartNew();
var result = await _client.GenerateAsync(
selectedModel,
prompt);
modelTimer.Stop();
return new BenchmarkResult(
strategy,
"general",
selectedModel,
routingLatency,
modelTimer.Elapsed.TotalMilliseconds,
result.InputTokens,
result.OutputTokens,
result.EstimatedCost,
result.Success);
}
}
The important design principle is that both strategies should execute the same workload.
Only the routing strategy should change.
Controlling Benchmark Variables
A fair benchmark requires controlled variables.
Keep the following consistent where possible:
Otherwise, differences can be attributed incorrectly to semantic routing.
For example, if static selection runs at concurrency 10 while semantic routing runs at concurrency 50, the comparison is not meaningful.
Benchmarking Under Concurrency
Routing behavior can change significantly under load.
Run multiple concurrency levels:
1
5
10
25
50
100
The exact levels should reflect your application.
Measure:
Average latency
Median latency
P95 latency
P99 latency
Error rate
Router saturation
Model distribution
Throughput
Tail latency is especially important.
A router that adds only a small average delay but causes large P99 spikes may still be unsuitable for interactive applications.
Comparing the Results
A useful result table might look like this:
| Metric | Static | Semantic Routing |
|---|
| Average latency | 1,850 ms | 1,420 ms |
| P95 latency | 3,400 ms | 2,900 ms |
| Average cost | $0.012 | $0.008 |
| Routing accuracy | N/A | 92% |
| Error rate | 1.2% | 1.5% |
| Under-routing | N/A | 4% |
| Over-routing | N/A | 7% |
These numbers are illustrative rather than universal benchmark results.
The decision should be based on measurements from your own workload.
Semantic Router Failure Modes
A production router needs failure handling.
Consider this scenario:
Request
|
v
Router
|
X
Router unavailable
The application should not necessarily fail every request.
A fallback policy can be used:
var model = "balanced-model";
try
{
var route = await router.SelectModelAsync(prompt);
model = route.Model;
}
catch
{
// Use a safe default model.
}
In production code, use structured exception handling and observability rather than silently swallowing errors.
The fallback model should be selected deliberately based on the application's reliability and quality requirements.
Common Mistakes
Comparing Only Model Latency
This ignores routing overhead.
Measuring Only Average Latency
P95 and P99 often reveal problems hidden by averages.
Ignoring Quality
A cheaper response is not an optimization if it fails the task.
Using an Unrealistically Simple Dataset
A router can look excellent on obvious prompts and fail on ambiguous production requests.
Changing Multiple Variables
If model configuration, prompts, concurrency, and routing logic all change simultaneously, the benchmark becomes difficult to interpret.
Ignoring Router Failures
The router is now part of the production request path and needs its own reliability strategy.
Treating Routing Accuracy as the Final Metric
The ultimate goal is application-level improvement, not classification accuracy by itself.
Best Practices
Establish a static-model baseline first.
Benchmark the entire request path.
Measure routing latency separately.
Track cost per successful task, not only cost per request.
Measure P50, P95, and P99 latency.
Evaluate both over-routing and under-routing.
Include ambiguous prompts in the benchmark dataset.
Use production-like workloads whenever possible.
Test routing behavior under concurrency.
Define a deterministic fallback model.
Monitor route distribution after deployment.
Re-evaluate routing rules when models or workloads change.
When Semantic Routing Makes Sense
Semantic routing is most attractive when workloads have meaningful differences in complexity.
For example:
80% Simple Requests
15% Medium Requests
5% Complex Requests
If most requests can be handled successfully by lower-cost models, routing can potentially reduce overall cost while preserving quality for difficult tasks.
On the other hand, if almost every request requires the same model capability, the router may simply add complexity without producing meaningful benefits.
Static selection can remain the better architecture when:
Workloads are highly predictable
Model requirements are nearly identical
Latency is extremely sensitive
Routing cost is significant
Operational simplicity is a priority
Frequently Asked Questions
Is semantic routing always cheaper?
No. The router introduces its own cost, and incorrect routing can increase both model usage and retries.
Does semantic routing always reduce latency?
No. A routing stage adds latency. The overall system becomes faster only when the reduction in downstream model latency outweighs routing overhead.
Should routing use another LLM?
It can, but that is not always the best choice. A lightweight classifier, embeddings, deterministic rules, or a hybrid approach may provide lower latency and cost.
What is the most important routing metric?
There is no single universal metric. For production systems, cost per successful task, quality, tail latency, routing accuracy, and failure rate should generally be considered together.
How should the fallback model be selected?
Choose a model that provides acceptable quality across the majority of workloads while maintaining the reliability and latency requirements of the application.
Conclusion
Semantic routing can make a multi-model .NET application more efficient, but it should be treated as an optimization that needs evidence rather than an automatic architectural improvement.
The correct comparison is not simply:
Model A vs Model B
It is:
Static Strategy
vs
Routing + Selected Model
That means the benchmark must include routing overhead, model latency, token usage, cost, quality, routing accuracy, failure rates, and tail latency.
The most useful result is therefore not a single benchmark number. It is a workload-level view showing where semantic routing creates measurable value and where static model selection remains the simpler and more reliable option.
A good routing architecture should earn its complexity through measurable improvements in cost, latency, quality, or reliability.