Introduction
As AI applications move from prototypes to production, routing becomes an important engineering problem.
A simple application may send every request to one model:
Application
|
v
AI Model
That architecture works until the model becomes unavailable, too expensive, too slow, or unsuitable for a particular request.
A production application may instead use several providers or models:
+--> Model A
|
Application --> Router
|
+--> Model B
|
+--> Model C
The difficult question is not simply which model should be used. It is also why the router selected that model and what happens when the selected model cannot complete the request.
This leads to two different routing strategies:
Semantic routing: choose a model based on the meaning, complexity, or characteristics of the request.
Failover routing: use an alternate model when the preferred model fails.
Microsoft.Extensions.AI provides abstractions that make it easier to build provider-independent AI applications and compose AI services. In a multi-provider application, those abstractions can be used as the foundation for routing, fallback, telemetry, and resilience patterns.
The important engineering insight is that semantic routing and failover routing solve different problems. Semantic routing optimizes model selection, while failover routing optimizes availability.
A production system often needs both.
What Is Semantic Routing?
Semantic routing evaluates the request before selecting an AI model.
For example:
User Request
|
v
Classify Request
|
+---- Simple question ----> Fast Model
|
+---- Coding task --------> Coding Model
|
+---- Complex reasoning --> Reasoning Model
The router might consider:
Request category
Prompt complexity
Required context size
Coding requirements
Expected reasoning depth
Latency requirements
Cost constraints
Output format
User tier
A simple routing policy might look like:
if request is simple:
use fast model
else if request is coding-related:
use coding model
else:
use reasoning model
The advantage is that every request does not automatically consume the most expensive model.
What Is Failover Routing?
Failover routing starts with a preferred provider or model and switches to another when the first option cannot serve the request.
Request
|
v
Primary Model
|
X
Failure
|
v
Fallback Model
|
v
Response
Failures may include:
The important distinction is that failover does not necessarily determine which model is best for the request.
It determines what should happen when the preferred path fails.
Semantic Routing vs Failover Routing
| Characteristic | Semantic Routing | Failover Routing |
|---|
| Main goal | Better model selection | Higher availability |
| Decision time | Before execution | During failure recovery |
| Primary input | Request characteristics | Runtime failure |
| Optimization | Cost, quality, latency | Reliability |
| Requires classification | Usually | No |
| Handles outages | Indirectly | Directly |
| Can reduce cost | Yes | Sometimes |
| Can improve availability | Sometimes | Yes |
These strategies should not be treated as competitors.
They can operate together.
A Combined Architecture
A production architecture can look like this:
User Request
|
v
Semantic Router
|
+---------------+---------------+
| | |
v v v
Fast Model Coding Model Reasoning Model
| | |
+---------------+---------------+
|
Failure?
|
+----+----+
| |
No Yes
| |
v v
Response Failover
|
v
Backup Model
This separates two responsibilities:
Semantic Router = Which model should handle this?
Failover Router = What should happen if it cannot?
That separation makes the system easier to test.
Building a Routing Abstraction
A simple .NET application can start with a routing interface:
public interface IAIRouter
{
Task<IChatClient> SelectClientAsync(
string request,
CancellationToken cancellationToken = default);
}
A semantic implementation could classify the request:
public sealed class SemanticAIRouter : IAIRouter
{
private readonly IChatClient _fastClient;
private readonly IChatClient _reasoningClient;
public SemanticAIRouter(
IChatClient fastClient,
IChatClient reasoningClient)
{
_fastClient = fastClient;
_reasoningClient = reasoningClient;
}
public Task<IChatClient> SelectClientAsync(
string request,
CancellationToken cancellationToken = default)
{
var client = request.Length > 1000
? _reasoningClient
: _fastClient;
return Task.FromResult(client);
}
}
This example is intentionally simple.
Production semantic routing should not rely only on prompt length. It could combine classification, historical performance, request type, token estimates, and business rules.
Adding Failover
The next layer can handle runtime failures:
public async Task<ChatResponse> ExecuteWithFailoverAsync(
IChatClient primary,
IChatClient fallback,
IEnumerable<ChatMessage> messages,
CancellationToken cancellationToken)
{
try
{
return await primary.GetResponseAsync(
messages,
cancellationToken: cancellationToken);
}
catch (Exception)
{
return await fallback.GetResponseAsync(
messages,
cancellationToken: cancellationToken);
}
}
The important production improvement is to avoid catching every exception blindly.
For example, an authentication error may not be recoverable by retrying another model.
A better design categorizes failures.
Failure Classification
Consider:
Request Failure
|
+--------------+--------------+
| | |
Retryable Provider Permanent
Failure Failure Failure
| | |
v v v
Retry Failover Return
Examples:
| Failure | Likely Action |
|---|
| Timeout | Retry or failover |
| Rate limit | Backoff or failover |
| Temporary service error | Failover |
| Invalid API key | Alert/configuration failure |
| Invalid request | Do not failover blindly |
| Content policy rejection | Apply application policy |
| Malformed response | Retry/failover depending on provider |
This is where routing becomes a resilience problem rather than a simple try/catch.
Benchmarking Semantic Routing
A routing benchmark should measure whether the router actually makes better decisions.
Useful metrics include:
Classification accuracy
Model-selection accuracy
Average latency
Cost per request
Quality score
Failure rate
Fallback frequency
Token consumption
Consider a test set of 1,000 requests.
The benchmark should record:
Request ID
Request category
Expected model
Selected model
Latency
Input tokens
Output tokens
Cost
Quality score
Fallback triggered
This allows you to answer questions such as:
Is semantic routing actually saving money without reducing answer quality?
Benchmarking Failover Routing
Failover requires a different test methodology.
Instead of testing only successful requests, intentionally introduce failures.
For example:
Test 1:
Primary available
Test 2:
Primary timeout
Test 3:
Primary rate limited
Test 4:
Primary unavailable
Test 5:
Primary returns malformed response
Then measure:
Primary success rate
Fallback success rate
Total recovery latency
User-visible failures
Fallback percentage
A Simple Benchmark Matrix
| Scenario | Primary | Fallback | Expected Result |
|---|
| Normal | Available | Available | Primary response |
| Timeout | Timeout | Available | Fallback response |
| Rate limit | Limited | Available | Fallback response |
| Outage | Down | Available | Fallback response |
| Both unavailable | Down | Down | Controlled failure |
| Invalid request | Rejects | Available | Usually no blind failover |
The last scenario is particularly important.
A fallback model cannot fix a fundamentally invalid request.
Measuring Latency
Suppose a request normally takes:
Primary:
800 ms
A failed primary might produce:
Primary timeout:
2000 ms
Fallback:
900 ms
Total:
2900 ms
The fallback successfully improves availability, but the user still experienced substantially higher latency.
This is why fallback success rate alone is not enough.
Track:
Normal latency
+
Failure detection latency
+
Fallback latency
=
Recovery latency
Measuring Cost
Semantic routing can reduce cost by sending simple requests to cheaper models.
For example, imagine:
100 requests
70 simple requests -> Model A
20 coding requests -> Model B
10 complex requests -> Model C
Instead of:
100 requests -> Model C
The actual savings depend on model pricing and token usage, so the benchmark should use real application telemetry rather than assumed numbers.
A useful metric is:
Average Cost Per Request
alongside:
Average Quality Score
Cost reduction without acceptable quality is not a successful routing strategy.
The Quality-Cost Frontier
A useful way to visualize semantic routing is:
Quality
^
|
| Model C
| *
|
| Model B
| *
|
| Model A
| *
+--------------------------> Cost
The router attempts to select a point that is appropriate for the request rather than always selecting the highest-cost model.
This creates a practical optimization problem:
Minimize cost
subject to:
quality >= required threshold
latency <= required threshold
availability >= required threshold
Semantic Routing Can Be Wrong
A semantic router is itself another source of failure.
Suppose the request is:
Explain why this distributed transaction implementation is failing.
The router might classify it as a simple programming question.
But the request may actually require:
If the router selects a lightweight model, the final response may be poor.
Therefore, benchmark the router itself.
Router Accuracy
A useful test set might contain:
Simple questions
Coding tasks
Debugging tasks
Architecture questions
Long-context requests
Reasoning-heavy problems
Structured-output requests
For each request:
Expected Route
Actual Route
Correct?
Then calculate:
Routing Accuracy =
Correct Routes / Total Requests
The benchmark should also evaluate the quality of incorrectly routed requests.
A router that achieves high classification accuracy but occasionally makes extremely expensive mistakes may still be unsuitable for production.
Avoiding Routing Loops
A poorly designed failover system can accidentally create loops.
For example:
Model A
|
v
Model B
|
v
Model A
|
v
Model B
Always define a finite fallback chain:
Primary
|
v
Fallback 1
|
v
Fallback 2
|
v
Controlled Failure
Set a maximum number of attempts.
Retry vs Failover
Retrying and failing over are also different.
Retry:
Same model
|
v
Try again
Failover:
Model A
|
X
|
Model B
A temporary network failure may justify retrying the same provider.
A provider outage should probably move to another provider.
The decision should be based on failure classification.
Observability
Routing decisions should be visible in telemetry.
A useful log entry might contain:
RequestId
SelectedRoute
Model
Provider
RoutingReason
Latency
RetryCount
FallbackTriggered
FallbackModel
InputTokens
OutputTokens
Success
For example:
RequestId: 84d2
Route: Coding
Primary: ProviderA
Fallback: ProviderB
FallbackTriggered: true
LatencyMs: 2410
Success: true
This makes production incidents significantly easier to investigate.
Common Mistakes
Treating Routing as a Static Rule
A rule such as:
if prompt.Length > 1000
is not a complete semantic router.
Prompt length is only one signal.
Using Failover for Every Error
Not every error is recoverable.
Invalid requests should not automatically be sent to another provider.
Ignoring Cost
A fallback strategy that always chooses the most expensive model can preserve availability while destroying cost efficiency.
Measuring Only Availability
A system can have excellent availability but poor quality or excessive latency.
Not Testing the Router
The routing layer needs its own evaluation dataset.
Allowing Unlimited Retries
Unlimited retry/failover behavior can increase latency, cost, and provider load.
Best Practices
Separate Selection From Recovery
Keep these responsibilities distinct:
Semantic Routing
|
v
Model Selection
|
v
Execution
|
v
Failure Handling
|
v
Failover
Use Explicit Policies
Define:
Which models handle which requests
Which failures trigger retries
Which failures trigger failover
Maximum attempts
Maximum latency
Cost limits
Benchmark With Real Workloads
Synthetic prompts are useful, but production-like requests provide better routing signals.
Track Quality and Cost Together
A routing strategy should optimize the entire operating point rather than a single metric.
Add Circuit Breaking for Persistent Provider Failures
If a provider repeatedly fails, temporarily stop sending traffic to it rather than discovering the outage independently on every request.
Keep Fallback Chains Short
A small number of well-tested alternatives is usually easier to operate than a large routing graph.
Advantages and Disadvantages
Semantic Routing Advantages
Can reduce model cost.
Can improve latency.
Matches models to workload characteristics.
Enables workload-specific optimization.
Makes multi-model architectures practical.
Semantic Routing Disadvantages
Router classification can be wrong.
Adds routing latency.
Requires evaluation.
More complex than a single model.
Routing logic requires maintenance.
Failover Routing Advantages
Improves availability.
Protects against provider outages.
Provides resilience during rate limiting.
Can reduce user-visible failures.
Failover Routing Disadvantages
Recovery increases latency.
May increase cost.
Incorrect failure classification can produce unnecessary fallbacks.
Multiple providers increase operational complexity.
Final Thoughts
Semantic routing and failover routing should not be viewed as competing approaches. They address different layers of a production AI system.
Semantic routing answers: "Which model is appropriate for this request?"
Failover routing answers: "What should happen when that model cannot serve the request?"
Microsoft.Extensions.AI provides the abstraction layer needed to build provider-independent AI applications, while routing and resilience policies can be composed around those AI clients.
The strongest architecture combines both approaches: classify the workload, select an appropriate model, execute the request, detect retryable failures, and move to a controlled fallback when necessary.
The real benchmark is not simply whether the system produces an answer. A production routing strategy should demonstrate that it can maintain an acceptable balance between quality, cost, latency, and availability under both normal traffic and failure conditions.