Microsoft.Extensions.AI Routing: Measuring Wrong-Model Selection Costs

Introduction

Adding multiple AI models to an application creates a new engineering problem: model selection.

A simple AI application can send every request to the same model:

User
 |
 v
Application
 |
 v
One AI Model
 |
 v
Response

A production application often has more choices:

                    +--> Fast Model
                    |
Application --> Router --> Coding Model
                    |
                    +--> Reasoning Model
                    |
                    +--> Low-Cost Model

At that point, the application needs to decide which model should handle each request.

That decision has a measurable cost.

Choosing a model that is too powerful can waste money and increase latency. Choosing a model that is too weak can reduce answer quality, cause additional retries, increase human intervention, or require the request to be processed again.

This article focuses on one specific problem in multi-model AI systems:

What does a wrong-model selection actually cost?

Microsoft.Extensions.AI provides common abstractions for working with AI services in .NET applications. These abstractions make it possible to build routing and provider-independent application layers without coupling every part of the application to a specific AI provider.

The goal here is not to assume that one model is always better than another. The goal is to create a measurable framework for understanding the consequences of selecting the wrong model.

What Is Wrong-Model Selection?

Suppose an application has three models:

Model A -> Fast and inexpensive
Model B -> Balanced
Model C -> More capable and expensive

Now consider this request:

Explain why this distributed transaction occasionally produces inconsistent state and propose a production-safe solution.

If the router sends this to Model A:

Request
  |
  v
Model A
  |
  v
Low-quality analysis

the application may have to retry with Model C.

That creates additional cost.

The opposite problem also exists.

A simple request such as:

Convert 25 Celsius to Fahrenheit.

does not normally require the most expensive reasoning model.

If the application sends every request to Model C:

Simple Request
     |
     v
Expensive Model

the answer may be correct, but the architecture is inefficient.

Two Types of Wrong Decisions

Wrong-model selection usually falls into two categories.

Under-Selection

The router chooses a model that is not capable enough for the task.

Complex Request
      |
      v
Weak Model
      |
      v
Poor Result

Possible consequences:

Lower answer quality
Additional retries
Human escalation
More tokens consumed
Longer end-to-end latency

Over-Selection

The router chooses a model that is more capable than necessary.

Simple Request
      |
      v
Expensive Model
      |
      v
Correct Result

The answer may be perfectly acceptable, but the application pays more than necessary.

This distinction is critical when measuring routing performance.

A Better Definition of Routing Cost

Model cost should not be reduced to API price alone.

A more useful equation is:

Total Routing Cost
=
Model Cost
+
Latency Cost
+
Retry Cost
+
Failure Cost
+
Human Intervention Cost

Not every organization can directly assign a monetary value to all these components, but the framework is useful.

For example:

Model Cost:
$0.02

Retry Cost:
$0.02

Additional Latency:
1.8 seconds

Human Review:
Required

The original routing decision may have looked cheap, but the total workflow cost is much higher.

Building a Routing Benchmark

Before optimizing routing, create a representative evaluation dataset.

A useful dataset might contain:

Simple Questions
Coding Tasks
Debugging Tasks
Architecture Questions
Summarization
Extraction
Long-Context Analysis
Reasoning Tasks
Structured Output

For every request, record:

Request ID
Request Category
Expected Model
Selected Model
Input Tokens
Output Tokens
Latency
Quality Score
Retry Count
Fallback Count
Estimated Cost

This turns routing from an intuition-based decision into an engineering measurement problem.

Establishing a Baseline

Start by sending every request to one strong model.

All Requests
     |
     v
Strong Model

Measure:

Average latency
Average cost
Quality
Failure rate
Token usage

This becomes the baseline.

Then introduce routing:

Requests
   |
   v
Router
   |
   +--> Model A
   +--> Model B
   +--> Model C

Compare the results.

Without a baseline, it is difficult to determine whether routing actually improved the system.

Defining the Correct Model

The hardest part of the benchmark is deciding what the "correct" model actually means.

There is rarely a single objectively correct answer.

Instead, define a minimum acceptable quality threshold.

For example:

Quality >= 0.85
Latency <= 2 seconds
Cost <= target

A model is considered suitable if it satisfies the application's requirements.

This creates a better optimization problem:

Choose the least expensive model
that satisfies the quality requirement.

Model Selection Matrix

You can define an initial routing policy using a matrix.

Workload	Preferred Model	Reason
Simple Q&A	Fast model	Low latency
Basic extraction	Low-cost model	Predictable workload
Coding	Coding-capable model	Code quality
Complex debugging	Reasoning model	Deeper analysis
Architecture review	Strong reasoning model	Complex tradeoffs
Large-context analysis	Long-context model	Context requirements

This is only a starting point.

Production telemetry should eventually determine whether these assumptions are correct.

Measuring Under-Selection Cost

Under-selection occurs when a request is routed to a model that cannot reliably satisfy the quality threshold.

Suppose:

100 complex requests

are routed to a lightweight model.

After evaluation:

78 acceptable
22 unacceptable

Those 22 requests may require:

Retry
   |
   v
Stronger Model

Now the effective workflow becomes:

Original Request
      |
      v
Weak Model
      |
      X
Poor Result
      |
      v
Strong Model
      |
      v
Final Result

The application has effectively paid for two inference attempts.

Measuring Over-Selection Cost

Over-selection is easier to overlook.

Imagine:

100 simple requests

All are routed to a powerful model.

The quality may be excellent.

But suppose a cheaper model could have achieved the same acceptable quality for 95 of those requests.

Then the application is paying a premium without receiving additional value.

A useful metric is:

Over-Selection Rate
=
Requests sent to unnecessarily expensive models
/
Total requests

This is one of the most useful metrics for cost optimization.

Quality-Cost Curves

For each model, calculate:

Average Quality
vs
Average Cost

Conceptually:

Quality
  ^
  |
  |                     Model C
  |                  *
  |
  |            Model B
  |         *
  |
  |    Model A
  |  *
  +----------------------------> Cost

The objective is not necessarily to select the model with the highest quality.

It is to select the model that provides sufficient quality at the lowest appropriate cost.

Quality Scoring

Quality can be evaluated using several approaches:

Human Evaluation

Developers or domain experts score responses.

Example:

1 = Incorrect
2 = Poor
3 = Acceptable
4 = Good
5 = Excellent

Automated Evaluation

A separate evaluator model can assess:

Correctness
Relevance
Completeness
Instruction following
Code validity

Deterministic Evaluation

For tasks with known outputs, use exact or structured comparisons.

For example:

JSON schema validation
Unit tests
Compilation
SQL execution
Mathematical verification

For software-development workloads, deterministic evaluation is particularly valuable.

Measuring Code Generation Routing

Consider a coding request:

Implement an asynchronous retry policy in C# with cancellation support.

A benchmark could evaluate the generated solution using:

Compile
   |
   v
Unit Tests
   |
   v
Static Analysis
   |
   v
Quality Score

Now routing quality is not based only on whether the model produced text.

It can be measured using actual engineering outcomes.

Example Benchmark

Consider three models:

Model A
Fast
Low cost

Model B
Balanced

Model C
High capability
High cost

A test suite contains:

40 simple tasks
40 coding tasks
20 reasoning tasks

A routing strategy might produce:

Simple:
Model A

Coding:
Model B

Reasoning:
Model C

Now compare that against:

Baseline:
All requests -> Model C

Measure:

Total cost
Average latency
Quality
Failure rate

The result might show that routing significantly reduces cost while maintaining the quality threshold.

The numbers should come from actual measurements rather than assumed percentages.

Measuring Wrong-Model Penalty

One useful metric is the wrong-model penalty.

Define:

Wrong-Model Penalty
=
Actual Cost of Selected Model
-
Cost of Best Acceptable Model

For example:

Selected Model:
$0.08

Best acceptable model:
$0.03

Penalty:
$0.05

For 10,000 requests:

$0.05 × 10,000 = $500

This illustrates why apparently small routing mistakes become significant at scale.

Quality Penalty

Cost is only one side of the problem.

For under-selection:

Quality Penalty
=
Required Quality
-
Actual Quality

For example:

Required:
0.90

Actual:
0.72

Quality gap:
0.18

This can be more important than the direct model cost.

A cheap model that produces unacceptable output is not actually cheap.

Expected Routing Cost

A more useful production metric is expected cost.

Consider:

P(model selected)
×
Model cost

For multiple models:

Expected Cost
=
Σ Probability(Model i) × Cost(Model i)

You can extend the equation to include retry probability:

Expected Cost
=
Primary Cost
+
Retry Probability × Retry Cost

And further:

Expected Total Cost
=
Inference Cost
+
Retry Cost
+
Fallback Cost

This allows routing policies to be compared mathematically.

Latency Has a Similar Problem

A model can be cheap but slow.

Another model can be expensive but much faster.

Therefore, routing should track:

Cost
Quality
Latency

together.

A useful dashboard might show:

Model	Quality	Avg Latency	Cost	Failure Rate
Model A	0.82	700 ms	Low	Low
Model B	0.90	1.2 s	Medium	Low
Model C	0.95	2.4 s	High	Low

The values above are illustrative benchmark fields, not universal model characteristics.

Routing Errors Under Production Load

Routing decisions can change under real workloads.

For example:

Normal Traffic
     |
     v
Model A

But during a traffic spike:

Traffic Spike
     |
     v
Model A Rate Limits
     |
     v
Fallback Model

Therefore, routing benchmarks should include load testing.

Measure:

Low traffic
Medium traffic
High traffic
Provider degradation
Provider outage

Combining Routing With Failover

Routing and failover work well together.

Semantic Router
      |
      v
Preferred Model
      |
      X
Failure
      |
      v
Fallback Model

But the fallback should still satisfy the application's minimum quality requirements.

Otherwise, availability improves while quality collapses.

Logging Routing Decisions

Every production request should ideally capture the routing decision.

Example:

logger.LogInformation(
    "AI route selected. Category={Category}, Model={Model}",
    requestCategory,
    selectedModel);

For a richer telemetry record:

RequestId
Category
SelectedModel
FallbackModel
RoutingReason
Latency
Tokens
EstimatedCost
QualityScore
RetryCount
Success

This information allows you to answer:

Why was this request sent to this model?

That question becomes extremely valuable during production incidents.

Common Mistakes

Optimizing Only for Cost

The cheapest model is not necessarily the cheapest workflow.

Poor quality can create retries and human intervention.

Optimizing Only for Quality

Always selecting the strongest model can make the system unnecessarily expensive.

Using Prompt Length as the Only Signal

Prompt length does not necessarily indicate reasoning complexity.

Ignoring Fallback Costs

A router may look inexpensive until fallback traffic is included.

Evaluating With Only One Task Type

A routing strategy trained around simple questions may fail badly on debugging or architecture tasks.

Ignoring Distribution Changes

If production traffic changes, routing assumptions can become outdated.

Best Practices

Build a Representative Evaluation Set

Use real workload categories.

Define Minimum Quality Thresholds

Do not ask:

Which model is best?

Ask:

Which is the least expensive model that satisfies this task's requirements?

Measure Wrong Decisions

Track both:

Under-selection
Over-selection

Include Retry and Fallback Costs

Measure the entire workflow.

Re-Evaluate Routing Periodically

Model capabilities and workload patterns change.

Use Deterministic Tests Where Possible

Compilation, unit tests, schema validation, and exact-match evaluation are stronger than subjective scoring for suitable workloads.

Add Observability

Routing decisions should be visible in telemetry.

Advantages and Disadvantages

Advantages

Makes model selection measurable.
Helps reduce unnecessary inference cost.
Identifies quality failures caused by weak models.
Supports evidence-based routing policies.
Makes multi-model AI systems easier to optimize.

Disadvantages

Requires evaluation infrastructure.
Quality measurement can be difficult.
Routing itself introduces complexity.
Model capabilities can change over time.
A routing policy that works for one workload may not work for another.

Final Thoughts

Wrong-model selection is one of the hidden costs of multi-model AI applications. The problem is not simply choosing an expensive model when a cheaper one would work. It also includes selecting a model that is too weak, causing retries, fallbacks, poor output, or human intervention.

Microsoft.Extensions.AI provides a useful abstraction for building provider-independent AI applications in .NET, but the routing policy around those abstractions still needs to be engineered and measured.

The most practical approach is to build a representative evaluation dataset, establish a strong-model baseline, measure quality and cost for each model, and then calculate the penalty created by incorrect routing decisions.

A good AI router should not try to find the most powerful model for every request. It should find the least expensive model that reliably meets the application's quality, latency, and availability requirements.

That is where routing becomes an engineering discipline rather than a collection of hard-coded model-selection rules.