Introduction
Adding multiple AI models to an application creates a new engineering problem: model selection.
A simple AI application can send every request to the same model:
User
|
v
Application
|
v
One AI Model
|
v
Response
A production application often has more choices:
+--> Fast Model
|
Application --> Router --> Coding Model
|
+--> Reasoning Model
|
+--> Low-Cost Model
At that point, the application needs to decide which model should handle each request.
That decision has a measurable cost.
Choosing a model that is too powerful can waste money and increase latency. Choosing a model that is too weak can reduce answer quality, cause additional retries, increase human intervention, or require the request to be processed again.
This article focuses on one specific problem in multi-model AI systems:
What does a wrong-model selection actually cost?
Microsoft.Extensions.AI provides common abstractions for working with AI services in .NET applications. These abstractions make it possible to build routing and provider-independent application layers without coupling every part of the application to a specific AI provider.
The goal here is not to assume that one model is always better than another. The goal is to create a measurable framework for understanding the consequences of selecting the wrong model.
What Is Wrong-Model Selection?
Suppose an application has three models:
Model A -> Fast and inexpensive
Model B -> Balanced
Model C -> More capable and expensive
Now consider this request:
Explain why this distributed transaction occasionally produces inconsistent state and propose a production-safe solution.
If the router sends this to Model A:
Request
|
v
Model A
|
v
Low-quality analysis
the application may have to retry with Model C.
That creates additional cost.
The opposite problem also exists.
A simple request such as:
Convert 25 Celsius to Fahrenheit.
does not normally require the most expensive reasoning model.
If the application sends every request to Model C:
Simple Request
|
v
Expensive Model
the answer may be correct, but the architecture is inefficient.
Two Types of Wrong Decisions
Wrong-model selection usually falls into two categories.
Under-Selection
The router chooses a model that is not capable enough for the task.
Complex Request
|
v
Weak Model
|
v
Poor Result
Possible consequences:
Over-Selection
The router chooses a model that is more capable than necessary.
Simple Request
|
v
Expensive Model
|
v
Correct Result
The answer may be perfectly acceptable, but the application pays more than necessary.
This distinction is critical when measuring routing performance.
A Better Definition of Routing Cost
Model cost should not be reduced to API price alone.
A more useful equation is:
Total Routing Cost
=
Model Cost
+
Latency Cost
+
Retry Cost
+
Failure Cost
+
Human Intervention Cost
Not every organization can directly assign a monetary value to all these components, but the framework is useful.
For example:
Model Cost:
$0.02
Retry Cost:
$0.02
Additional Latency:
1.8 seconds
Human Review:
Required
The original routing decision may have looked cheap, but the total workflow cost is much higher.
Building a Routing Benchmark
Before optimizing routing, create a representative evaluation dataset.
A useful dataset might contain:
Simple Questions
Coding Tasks
Debugging Tasks
Architecture Questions
Summarization
Extraction
Long-Context Analysis
Reasoning Tasks
Structured Output
For every request, record:
Request ID
Request Category
Expected Model
Selected Model
Input Tokens
Output Tokens
Latency
Quality Score
Retry Count
Fallback Count
Estimated Cost
This turns routing from an intuition-based decision into an engineering measurement problem.
Establishing a Baseline
Start by sending every request to one strong model.
All Requests
|
v
Strong Model
Measure:
Average latency
Average cost
Quality
Failure rate
Token usage
This becomes the baseline.
Then introduce routing:
Requests
|
v
Router
|
+--> Model A
+--> Model B
+--> Model C
Compare the results.
Without a baseline, it is difficult to determine whether routing actually improved the system.
Defining the Correct Model
The hardest part of the benchmark is deciding what the "correct" model actually means.
There is rarely a single objectively correct answer.
Instead, define a minimum acceptable quality threshold.
For example:
Quality >= 0.85
Latency <= 2 seconds
Cost <= target
A model is considered suitable if it satisfies the application's requirements.
This creates a better optimization problem:
Choose the least expensive model
that satisfies the quality requirement.
Model Selection Matrix
You can define an initial routing policy using a matrix.
| Workload | Preferred Model | Reason |
|---|
| Simple Q&A | Fast model | Low latency |
| Basic extraction | Low-cost model | Predictable workload |
| Coding | Coding-capable model | Code quality |
| Complex debugging | Reasoning model | Deeper analysis |
| Architecture review | Strong reasoning model | Complex tradeoffs |
| Large-context analysis | Long-context model | Context requirements |
This is only a starting point.
Production telemetry should eventually determine whether these assumptions are correct.
Measuring Under-Selection Cost
Under-selection occurs when a request is routed to a model that cannot reliably satisfy the quality threshold.
Suppose:
100 complex requests
are routed to a lightweight model.
After evaluation:
78 acceptable
22 unacceptable
Those 22 requests may require:
Retry
|
v
Stronger Model
Now the effective workflow becomes:
Original Request
|
v
Weak Model
|
X
Poor Result
|
v
Strong Model
|
v
Final Result
The application has effectively paid for two inference attempts.
Measuring Over-Selection Cost
Over-selection is easier to overlook.
Imagine:
100 simple requests
All are routed to a powerful model.
The quality may be excellent.
But suppose a cheaper model could have achieved the same acceptable quality for 95 of those requests.
Then the application is paying a premium without receiving additional value.
A useful metric is:
Over-Selection Rate
=
Requests sent to unnecessarily expensive models
/
Total requests
This is one of the most useful metrics for cost optimization.
Quality-Cost Curves
For each model, calculate:
Average Quality
vs
Average Cost
Conceptually:
Quality
^
|
| Model C
| *
|
| Model B
| *
|
| Model A
| *
+----------------------------> Cost
The objective is not necessarily to select the model with the highest quality.
It is to select the model that provides sufficient quality at the lowest appropriate cost.
Quality Scoring
Quality can be evaluated using several approaches:
Human Evaluation
Developers or domain experts score responses.
Example:
1 = Incorrect
2 = Poor
3 = Acceptable
4 = Good
5 = Excellent
Automated Evaluation
A separate evaluator model can assess:
Correctness
Relevance
Completeness
Instruction following
Code validity
Deterministic Evaluation
For tasks with known outputs, use exact or structured comparisons.
For example:
JSON schema validation
Unit tests
Compilation
SQL execution
Mathematical verification
For software-development workloads, deterministic evaluation is particularly valuable.
Measuring Code Generation Routing
Consider a coding request:
Implement an asynchronous retry policy in C# with cancellation support.
A benchmark could evaluate the generated solution using:
Compile
|
v
Unit Tests
|
v
Static Analysis
|
v
Quality Score
Now routing quality is not based only on whether the model produced text.
It can be measured using actual engineering outcomes.
Example Benchmark
Consider three models:
Model A
Fast
Low cost
Model B
Balanced
Model C
High capability
High cost
A test suite contains:
40 simple tasks
40 coding tasks
20 reasoning tasks
A routing strategy might produce:
Simple:
Model A
Coding:
Model B
Reasoning:
Model C
Now compare that against:
Baseline:
All requests -> Model C
Measure:
Total cost
Average latency
Quality
Failure rate
The result might show that routing significantly reduces cost while maintaining the quality threshold.
The numbers should come from actual measurements rather than assumed percentages.
Measuring Wrong-Model Penalty
One useful metric is the wrong-model penalty.
Define:
Wrong-Model Penalty
=
Actual Cost of Selected Model
-
Cost of Best Acceptable Model
For example:
Selected Model:
$0.08
Best acceptable model:
$0.03
Penalty:
$0.05
For 10,000 requests:
$0.05 × 10,000 = $500
This illustrates why apparently small routing mistakes become significant at scale.
Quality Penalty
Cost is only one side of the problem.
For under-selection:
Quality Penalty
=
Required Quality
-
Actual Quality
For example:
Required:
0.90
Actual:
0.72
Quality gap:
0.18
This can be more important than the direct model cost.
A cheap model that produces unacceptable output is not actually cheap.
Expected Routing Cost
A more useful production metric is expected cost.
Consider:
P(model selected)
×
Model cost
For multiple models:
Expected Cost
=
Σ Probability(Model i) × Cost(Model i)
You can extend the equation to include retry probability:
Expected Cost
=
Primary Cost
+
Retry Probability × Retry Cost
And further:
Expected Total Cost
=
Inference Cost
+
Retry Cost
+
Fallback Cost
This allows routing policies to be compared mathematically.
Latency Has a Similar Problem
A model can be cheap but slow.
Another model can be expensive but much faster.
Therefore, routing should track:
Cost
Quality
Latency
together.
A useful dashboard might show:
| Model | Quality | Avg Latency | Cost | Failure Rate |
|---|
| Model A | 0.82 | 700 ms | Low | Low |
| Model B | 0.90 | 1.2 s | Medium | Low |
| Model C | 0.95 | 2.4 s | High | Low |
The values above are illustrative benchmark fields, not universal model characteristics.
Routing Errors Under Production Load
Routing decisions can change under real workloads.
For example:
Normal Traffic
|
v
Model A
But during a traffic spike:
Traffic Spike
|
v
Model A Rate Limits
|
v
Fallback Model
Therefore, routing benchmarks should include load testing.
Measure:
Low traffic
Medium traffic
High traffic
Provider degradation
Provider outage
Combining Routing With Failover
Routing and failover work well together.
Semantic Router
|
v
Preferred Model
|
X
Failure
|
v
Fallback Model
But the fallback should still satisfy the application's minimum quality requirements.
Otherwise, availability improves while quality collapses.
Logging Routing Decisions
Every production request should ideally capture the routing decision.
Example:
logger.LogInformation(
"AI route selected. Category={Category}, Model={Model}",
requestCategory,
selectedModel);
For a richer telemetry record:
RequestId
Category
SelectedModel
FallbackModel
RoutingReason
Latency
Tokens
EstimatedCost
QualityScore
RetryCount
Success
This information allows you to answer:
Why was this request sent to this model?
That question becomes extremely valuable during production incidents.
Common Mistakes
Optimizing Only for Cost
The cheapest model is not necessarily the cheapest workflow.
Poor quality can create retries and human intervention.
Optimizing Only for Quality
Always selecting the strongest model can make the system unnecessarily expensive.
Using Prompt Length as the Only Signal
Prompt length does not necessarily indicate reasoning complexity.
Ignoring Fallback Costs
A router may look inexpensive until fallback traffic is included.
Evaluating With Only One Task Type
A routing strategy trained around simple questions may fail badly on debugging or architecture tasks.
Ignoring Distribution Changes
If production traffic changes, routing assumptions can become outdated.
Best Practices
Build a Representative Evaluation Set
Use real workload categories.
Define Minimum Quality Thresholds
Do not ask:
Which model is best?
Ask:
Which is the least expensive model that satisfies this task's requirements?
Measure Wrong Decisions
Track both:
Under-selection
Over-selection
Include Retry and Fallback Costs
Measure the entire workflow.
Re-Evaluate Routing Periodically
Model capabilities and workload patterns change.
Use Deterministic Tests Where Possible
Compilation, unit tests, schema validation, and exact-match evaluation are stronger than subjective scoring for suitable workloads.
Add Observability
Routing decisions should be visible in telemetry.
Advantages and Disadvantages
Advantages
Makes model selection measurable.
Helps reduce unnecessary inference cost.
Identifies quality failures caused by weak models.
Supports evidence-based routing policies.
Makes multi-model AI systems easier to optimize.
Disadvantages
Requires evaluation infrastructure.
Quality measurement can be difficult.
Routing itself introduces complexity.
Model capabilities can change over time.
A routing policy that works for one workload may not work for another.
Final Thoughts
Wrong-model selection is one of the hidden costs of multi-model AI applications. The problem is not simply choosing an expensive model when a cheaper one would work. It also includes selecting a model that is too weak, causing retries, fallbacks, poor output, or human intervention.
Microsoft.Extensions.AI provides a useful abstraction for building provider-independent AI applications in .NET, but the routing policy around those abstractions still needs to be engineered and measured.
The most practical approach is to build a representative evaluation dataset, establish a strong-model baseline, measure quality and cost for each model, and then calculate the penalty created by incorrect routing decisions.
A good AI router should not try to find the most powerful model for every request. It should find the least expensive model that reliably meets the application's quality, latency, and availability requirements.
That is where routing becomes an engineering discipline rather than a collection of hard-coded model-selection rules.