Introduction
Choosing an AI model is no longer just about finding the model that gives the best answer.
A production application may have several models available. One may be fast and inexpensive, another may be better at complex reasoning, and a third may provide a good balance between the two.
The challenge is deciding which model should handle which request.
Sending every request to the most capable model can work, but it can also increase cost and latency unnecessarily. Sending everything to the cheapest model can reduce cost while hurting answer quality.
Model routing tries to find a better middle ground.
Instead of using one model for every request, the application decides which model is appropriate for each workload.
User Request
↓
Router
↓
┌──────────────┬──────────────┬──────────────┐
│ Fast Model │ Balanced │ Reasoning │
│ │ Model │ Model │
└──────────────┴──────────────┴──────────────┘
The difficult part is proving that the router actually improves the application.
That requires measuring three things together:
Cost
Latency
Answer quality
A routing strategy that saves money but produces poor answers is not successful. A strategy that produces excellent answers but costs several times more may not be practical either.
What Is Model Routing?
Model routing is the process of selecting an AI model dynamically based on the characteristics of a request.
For example:
Simple question
↓
Fast / low-cost model
Complex reasoning
↓
More capable model
Code generation
↓
Code-specialized model
The router can use information such as:
A simple routing function might look like:
public string SelectModel(string request)
{
if (request.Length < 500)
{
return "fast-model";
}
return "reasoning-model";
}
This is easy to build, but it is not enough for production.
The real question is whether the routing decision improves the overall system.
Why Routing Needs Benchmarking
Imagine an application that currently uses one model:
100,000 requests
Average cost: $0.010
Average latency: 900 ms
Quality score: 4.6 / 5
A new router sends simple requests to a cheaper model:
Average cost: $0.006
Average latency: 500 ms
Quality score: 4.2 / 5
The router reduced cost and latency.
But it also reduced quality.
Whether that is acceptable depends on the application.
For a customer-support chatbot, a small quality reduction might be acceptable.
For code generation or financial analysis, it might not be.
This is why model routing should be treated as an optimization problem rather than simply a cost-saving technique.
Define the Three Metrics
Before building a router, define the metrics you want to optimize.
Cost
How much does each request cost?
Latency
How long does the user wait for the response?
Quality
How useful and correct is the response?
A simple evaluation table might look like:
| Model | Cost | P50 Latency | Quality |
|---|
| Fast | $0.002 | 300 ms | 3.9 |
| Balanced | $0.006 | 600 ms | 4.4 |
| Reasoning | $0.015 | 1.4 s | 4.8 |
There is no universally best model.
The best choice depends on the request.
Categorize Requests First
Before routing requests, divide your workload into meaningful categories.
For example:
Customer Support
Code Generation
Summarization
Classification
Data Extraction
Complex Reasoning
Then benchmark each model against each category.
You may discover something interesting.
For example:
| Task | Fast | Balanced | Reasoning |
|---|
| Classification | 4.5 | 4.6 | 4.7 |
| Summarization | 4.2 | 4.5 | 4.6 |
| Code Generation | 3.7 | 4.4 | 4.8 |
| Complex Reasoning | 3.2 | 4.1 | 4.8 |
The reasoning model may provide little additional value for classification but a large improvement for complex reasoning.
That is exactly where routing becomes useful.
Build a Representative Evaluation Dataset
Do not benchmark models using five manually selected prompts.
Create a representative dataset.
For example:
[
{
"id": "support-001",
"category": "support",
"input": "How can I change my billing address?"
},
{
"id": "code-001",
"category": "coding",
"input": "Write a C# method that retries an HTTP request."
}
]
The dataset should represent real application traffic as closely as possible.
Include:
Easy requests
Medium requests
Difficult requests
Short inputs
Long inputs
Edge cases
Failure cases
This prevents the router from being optimized for an unrealistic workload.
Run Every Model Against the Same Requests
For benchmarking, every candidate model should receive the same evaluation dataset.
Conceptually:
Test Dataset
|
+-----------+-----------+
| | |
Model A Model B Model C
| | |
+-----------+-----------+
↓
Evaluation
This makes the results comparable.
For each request, capture:
Model
Input Tokens
Output Tokens
Latency
Cost
Quality Score
Error
A result record could look like:
public sealed record ModelEvaluation(
string Model,
string Category,
double LatencyMs,
int InputTokens,
int OutputTokens,
decimal Cost,
double QualityScore);
Measure Latency Properly
Do not rely only on average latency.
Measure:
P50
P90
P95
P99
For interactive applications, P95 is especially useful.
Imagine:
Model A
P50 = 400 ms
P95 = 700 ms
Model B
P50 = 450 ms
P95 = 2,500 ms
The average could make these models look similar.
But users may experience Model B as much slower during peak or difficult requests.
Also separate:
Time to First Token
from:
Time to Complete Response
Streaming applications often care more about how quickly the first useful content appears.
Measure Cost Per Request
Cost should be calculated from actual token usage rather than simply assigning an average cost to every request.
For example:
Input tokens: 1,000
Output tokens: 300
If the model has different input and output prices, calculate them separately.
Conceptually:
Cost =
(input tokens × input price)
+
(output tokens × output price)
Your benchmark should record the actual token counts.
public sealed record UsageMetrics(
int InputTokens,
int OutputTokens,
decimal EstimatedCost);
This allows you to calculate the total cost of a routing strategy later.
Calculate Cost at the Workload Level
Suppose you have 10,000 requests.
Without routing:
10,000 × $0.010
= $100
With routing:
7,000 fast requests × $0.002
= $14
3,000 advanced requests × $0.015
= $45
Total = $59
The routing strategy reduces the estimated model cost from $100 to $59.
That is useful information.
But now evaluate quality.
If the quality score drops significantly, the savings may not justify the trade-off.
Measure Quality With More Than One Metric
"Answer quality" is not one universal measurement.
Depending on the application, evaluate:
For example, a code-generation benchmark might use:
Compilation: PASS
Unit tests: PASS
Security checks: PASS
Explanation score: 4.2/5
This is more useful than asking another model whether the generated code "looks good."
Use Deterministic Evaluation Where Possible
Some outputs can be evaluated without another AI model.
For example:
if (!result.IsValidJson)
{
return EvaluationResult.Fail(
"Invalid JSON output.");
}
For generated code:
Generate Code
↓
Compile
↓
Run Tests
↓
Security Checks
For classification:
Expected: "billing"
Actual: "billing"
These deterministic tests are more reliable than subjective evaluation.
Use LLM-based evaluation where objective checks are difficult.
Add an Evaluation Score
For tasks that require subjective evaluation, create a consistent scoring system.
For example:
Correctness 40%
Relevance 25%
Completeness 20%
Clarity 15%
Then calculate:
Final Score =
Correctness × 0.40
+ Relevance × 0.25
+ Completeness × 0.20
+ Clarity × 0.15
The exact weights depend on the application.
The important part is keeping the scoring method consistent across models.
Build a Simple Router
Once the benchmark data is available, the router can start with straightforward rules.
public string SelectModel(
string category,
int inputTokens)
{
if (category == "complex-reasoning")
{
return "reasoning-model";
}
if (inputTokens > 4000)
{
return "balanced-model";
}
return "fast-model";
}
This is simple, explainable, and easy to debug.
Start with rules before jumping into a machine-learning-based router.
Add Quality Thresholds
The router should not optimize cost without considering quality.
Suppose:
Minimum acceptable quality = 4.2
Then:
Fast Model
Quality = 3.9
Cost = $0.002
Balanced Model
Quality = 4.4
Cost = $0.006
The router should choose the balanced model for that task.
A simple decision function could be:
if (fast.QualityScore >= minimumQuality)
{
return fast;
}
return balanced;
The benchmark determines what "acceptable" actually means.
Use Cost and Latency Budgets
Some requests may have strict latency requirements.
For example:
Interactive request
Maximum latency: 1 second
A model with excellent quality but a typical latency of 2 seconds may not be suitable.
The router can apply constraints:
Quality >= 4.2
AND
P95 latency <= 1 second
Among the models that satisfy those conditions, select the lowest-cost option.
This is often a better strategy than simply selecting the cheapest model.
Think in Terms of Constraints
A practical routing policy can be expressed as:
1. Remove models that fail quality requirements.
2. Remove models that fail latency requirements.
3. Remove models that exceed cost limits.
4. Choose the best remaining model.
For example:
All Models
↓
Quality >= 4.2
↓
P95 <= 1 second
↓
Lowest Cost
↓
Selected
This is easier to reason about than a complicated scoring formula.
Example C# Router
A simple model definition might be:
public sealed record ModelOption(
string Name,
decimal Cost,
double P95LatencyMs,
double QualityScore);
Then:
public static ModelOption SelectModel(
IEnumerable<ModelOption> models)
{
return models
.Where(m => m.QualityScore >= 4.2)
.Where(m => m.P95LatencyMs <= 1000)
.OrderBy(m => m.Cost)
.First();
}
This is obviously simplified, but it demonstrates the basic idea.
The production version can include request-specific requirements.
Routing Based on Request Complexity
Not every request needs the same level of reasoning.
A router might classify requests:
Simple
Medium
Complex
Then map them:
Simple → Fast Model
Medium → Balanced Model
Complex → Reasoning Model
The complexity classifier itself needs to be evaluated.
If the classifier incorrectly sends 30% of difficult requests to the cheap model, the routing system may save money while significantly hurting quality.
The classifier is therefore part of the benchmark.
Test Routing Accuracy
Create an evaluation set with known categories:
Prompt 1 → Simple
Prompt 2 → Simple
Prompt 3 → Complex
Prompt 4 → Complex
Then compare:
Expected Route
Actual Route
A routing confusion matrix can be useful:
| Expected | Fast | Balanced | Reasoning |
|---|
| Simple | 94% | 5% | 1% |
| Medium | 12% | 80% | 8% |
| Complex | 2% | 15% | 83% |
This tells you where the router is making mistakes.
Benchmark the Router Itself
The router adds latency.
If model selection takes 200 ms and the chosen model takes 300 ms, the routing system has consumed a large portion of the request latency.
The router should therefore be lightweight.
Measure:
Routing latency
+
Model latency
+
Post-processing latency
For a simple rules-based router, routing should normally be very small compared with model inference.
Consider Fallback Routing
Models can fail.
A production router should define what happens when the selected model is unavailable.
For example:
Primary Model
↓
Failure
↓
Fallback Model
↓
Response
The fallback should also respect application constraints.
If the primary model fails, the system should not automatically select a model that violates security or cost requirements.
Avoid Endless Fallbacks
A poor fallback implementation can create unexpected costs.
For example:
Model A fails
↓
Model B fails
↓
Model C fails
↓
Model D fails
One user request could turn into four model requests.
Set a maximum retry or fallback budget.
const int maxAttempts = 2;
Also record every fallback event.
A high fallback rate may indicate a reliability problem rather than a routing opportunity.
Track Routing Decisions
Every request should ideally record:
Request ID
Task Category
Selected Model
Reason
Fallback
Latency
Token Usage
Cost
Quality
For example:
Request: req-123
Category: coding
Selected: balanced-model
Reason: complexity=medium
Fallback: none
Latency: 620 ms
Cost: $0.006
Quality: 4.5
This makes routing decisions explainable.
Compare Fixed and Dynamic Routing
Always compare the router against a baseline.
Fixed Model
Every request → Balanced Model
Dynamic Model
Simple → Fast
Medium → Balanced
Complex → Reasoning
Then compare:
| Metric | Fixed | Dynamic |
|---|
| Cost | $100 | $62 |
| P50 latency | 700 ms | 480 ms |
| P95 latency | 1.5 s | 1.1 s |
| Quality | 4.5 | 4.4 |
Now you have evidence that the routing strategy is useful.
Without the baseline, it is difficult to know whether routing actually improved the system.
Common Mistakes
Choosing the Cheapest Model for Everything
Cost is only one part of the decision.
Using Quality Scores Without a Dataset
A quality number is meaningless without representative test cases.
Ignoring P95 Latency
Average latency can hide slow requests.
Routing Only by Prompt Length
A short question can require complex reasoning, while a long question can be simple.
Ignoring Router Errors
A bad classifier can send difficult requests to weak models.
No Baseline
Always compare dynamic routing against a fixed-model strategy.
Unlimited Fallbacks
Fallbacks can unexpectedly multiply model usage and cost.
Using an LLM to Evaluate Everything
Use deterministic tests wherever possible.
Best Practices
Benchmark every candidate model on the same evaluation dataset.
Measure cost, latency, and quality together.
Use P50 and P95 latency instead of averages alone.
Measure time to first token for streaming applications.
Use deterministic quality checks whenever possible.
Establish a fixed-model baseline.
Add minimum quality requirements.
Add latency and cost constraints.
Start with simple, explainable routing rules.
Test the request classifier separately.
Measure router overhead.
Define controlled fallback behavior.
Record routing decisions.
Monitor real production routing results.
Periodically re-benchmark because model performance and pricing can change.
FAQs
Is model routing mainly about reducing cost?
No. Cost is one of the main reasons to use routing, but latency and answer quality are equally important.
Should I always send difficult requests to the most powerful model?
Not necessarily. First define what "difficult" means for your application and verify that the more capable model actually provides better results for those requests.
How do I know whether routing is working?
Compare the routed system with a fixed-model baseline using the same evaluation dataset. Look at total cost, latency distribution, quality, and failure rate.
Should routing use an AI model?
It can, but it does not have to. Start with simple rules or classifiers that are easy to understand and measure. More sophisticated routing can be introduced when the workload justifies it.
How often should routing benchmarks be updated?
Whenever the model, prompt, runtime, pricing, or application workload changes significantly. Production data should also be used to identify new routing cases for the evaluation dataset.
Conclusion
Model routing can make an AI application faster and more economical, but only when the routing decisions are based on real workload data. A cheap model is not useful if it consistently produces poor answers, and a powerful model may be unnecessary for simple requests. The practical approach is to benchmark candidate models using the same representative dataset, measure cost, latency, and quality together, establish a fixed-model baseline, and then introduce simple routing rules with clear quality and performance limits. Over time, production results can be added back into the evaluation dataset so the router improves based on real usage rather than assumptions. The goal is not to find one model that wins every benchmark; it is to send each request to a model that provides the right balance of quality, speed, and cost for that particular workload.