Benchmarking Semantic Routing vs Failover Routing in microsoft.Extensions.AI

Introduction

As AI applications move from prototypes to production, routing becomes an important engineering problem.

A simple application may send every request to one model:

Application
    |
    v
AI Model

That architecture works until the model becomes unavailable, too expensive, too slow, or unsuitable for a particular request.

A production application may instead use several providers or models:

                    +--> Model A
                    |
Application --> Router
                    |
                    +--> Model B
                    |
                    +--> Model C

The difficult question is not simply which model should be used. It is also why the router selected that model and what happens when the selected model cannot complete the request.

This leads to two different routing strategies:

Semantic routing: choose a model based on the meaning, complexity, or characteristics of the request.
Failover routing: use an alternate model when the preferred model fails.

Microsoft.Extensions.AI provides abstractions that make it easier to build provider-independent AI applications and compose AI services. In a multi-provider application, those abstractions can be used as the foundation for routing, fallback, telemetry, and resilience patterns.

The important engineering insight is that semantic routing and failover routing solve different problems. Semantic routing optimizes model selection, while failover routing optimizes availability.

A production system often needs both.

What Is Semantic Routing?

Semantic routing evaluates the request before selecting an AI model.

For example:

User Request
     |
     v
Classify Request
     |
     +---- Simple question ----> Fast Model
     |
     +---- Coding task --------> Coding Model
     |
     +---- Complex reasoning --> Reasoning Model

The router might consider:

Request category
Prompt complexity
Required context size
Coding requirements
Expected reasoning depth
Latency requirements
Cost constraints
Output format
User tier

A simple routing policy might look like:

if request is simple:
    use fast model

else if request is coding-related:
    use coding model

else:
    use reasoning model

The advantage is that every request does not automatically consume the most expensive model.

What Is Failover Routing?

Failover routing starts with a preferred provider or model and switches to another when the first option cannot serve the request.

Request
   |
   v
Primary Model
   |
   X
Failure
   |
   v
Fallback Model
   |
   v
Response

Failures may include:

Timeout
Rate limiting
Provider outage
Network failure
Service unavailable response
Temporary capacity problems

The important distinction is that failover does not necessarily determine which model is best for the request.

It determines what should happen when the preferred path fails.

Semantic Routing vs Failover Routing

Characteristic	Semantic Routing	Failover Routing
Main goal	Better model selection	Higher availability
Decision time	Before execution	During failure recovery
Primary input	Request characteristics	Runtime failure
Optimization	Cost, quality, latency	Reliability
Requires classification	Usually	No
Handles outages	Indirectly	Directly
Can reduce cost	Yes	Sometimes
Can improve availability	Sometimes	Yes

These strategies should not be treated as competitors.

They can operate together.

A Combined Architecture

A production architecture can look like this:

                         User Request
                              |
                              v
                     Semantic Router
                              |
              +---------------+---------------+
              |               |               |
              v               v               v
          Fast Model      Coding Model    Reasoning Model
              |               |               |
              +---------------+---------------+
                              |
                         Failure?
                              |
                         +----+----+
                         |         |
                        No        Yes
                         |         |
                         v         v
                      Response   Failover
                                   |
                                   v
                              Backup Model

This separates two responsibilities:

Semantic Router = Which model should handle this?

Failover Router  = What should happen if it cannot?

That separation makes the system easier to test.

Building a Routing Abstraction

A simple .NET application can start with a routing interface:

public interface IAIRouter
{
    Task<IChatClient> SelectClientAsync(
        string request,
        CancellationToken cancellationToken = default);
}

A semantic implementation could classify the request:

public sealed class SemanticAIRouter : IAIRouter
{
    private readonly IChatClient _fastClient;
    private readonly IChatClient _reasoningClient;

    public SemanticAIRouter(
        IChatClient fastClient,
        IChatClient reasoningClient)
    {
        _fastClient = fastClient;
        _reasoningClient = reasoningClient;
    }

    public Task<IChatClient> SelectClientAsync(
        string request,
        CancellationToken cancellationToken = default)
    {
        var client = request.Length > 1000
            ? _reasoningClient
            : _fastClient;

        return Task.FromResult(client);
    }
}

This example is intentionally simple.

Production semantic routing should not rely only on prompt length. It could combine classification, historical performance, request type, token estimates, and business rules.

Adding Failover

The next layer can handle runtime failures:

public async Task<ChatResponse> ExecuteWithFailoverAsync(
    IChatClient primary,
    IChatClient fallback,
    IEnumerable<ChatMessage> messages,
    CancellationToken cancellationToken)
{
    try
    {
        return await primary.GetResponseAsync(
            messages,
            cancellationToken: cancellationToken);
    }
    catch (Exception)
    {
        return await fallback.GetResponseAsync(
            messages,
            cancellationToken: cancellationToken);
    }
}

The important production improvement is to avoid catching every exception blindly.

For example, an authentication error may not be recoverable by retrying another model.

A better design categorizes failures.

Failure Classification

Consider:

                     Request Failure
                           |
            +--------------+--------------+
            |              |              |
          Retryable     Provider       Permanent
           Failure       Failure         Failure
            |              |              |
            v              v              v
          Retry          Failover        Return

Examples:

Failure	Likely Action
Timeout	Retry or failover
Rate limit	Backoff or failover
Temporary service error	Failover
Invalid API key	Alert/configuration failure
Invalid request	Do not failover blindly
Content policy rejection	Apply application policy
Malformed response	Retry/failover depending on provider

This is where routing becomes a resilience problem rather than a simple try/catch.

Benchmarking Semantic Routing

A routing benchmark should measure whether the router actually makes better decisions.

Useful metrics include:

Classification accuracy
Model-selection accuracy
Average latency
Cost per request
Quality score
Failure rate
Fallback frequency
Token consumption

Consider a test set of 1,000 requests.

The benchmark should record:

Request ID
Request category
Expected model
Selected model
Latency
Input tokens
Output tokens
Cost
Quality score
Fallback triggered

This allows you to answer questions such as:

Is semantic routing actually saving money without reducing answer quality?

Benchmarking Failover Routing

Failover requires a different test methodology.

Instead of testing only successful requests, intentionally introduce failures.

For example:

Test 1:
Primary available

Test 2:
Primary timeout

Test 3:
Primary rate limited

Test 4:
Primary unavailable

Test 5:
Primary returns malformed response

Then measure:

Primary success rate
Fallback success rate
Total recovery latency
User-visible failures
Fallback percentage

A Simple Benchmark Matrix

Scenario	Primary	Fallback	Expected Result
Normal	Available	Available	Primary response
Timeout	Timeout	Available	Fallback response
Rate limit	Limited	Available	Fallback response
Outage	Down	Available	Fallback response
Both unavailable	Down	Down	Controlled failure
Invalid request	Rejects	Available	Usually no blind failover

The last scenario is particularly important.

A fallback model cannot fix a fundamentally invalid request.

Measuring Latency

Suppose a request normally takes:

Primary:
800 ms

A failed primary might produce:

Primary timeout:
2000 ms

Fallback:
900 ms

Total:
2900 ms

The fallback successfully improves availability, but the user still experienced substantially higher latency.

This is why fallback success rate alone is not enough.

Track:

Normal latency
+
Failure detection latency
+
Fallback latency
=
Recovery latency

Measuring Cost

Semantic routing can reduce cost by sending simple requests to cheaper models.

For example, imagine:

100 requests

70 simple requests  -> Model A
20 coding requests  -> Model B
10 complex requests -> Model C

Instead of:

100 requests -> Model C

The actual savings depend on model pricing and token usage, so the benchmark should use real application telemetry rather than assumed numbers.

A useful metric is:

Average Cost Per Request

alongside:

Average Quality Score

Cost reduction without acceptable quality is not a successful routing strategy.

The Quality-Cost Frontier

A useful way to visualize semantic routing is:

Quality
  ^
  |
  |                  Model C
  |              *
  |
  |        Model B
  |      *
  |
  | Model A
  |   *
  +--------------------------> Cost

The router attempts to select a point that is appropriate for the request rather than always selecting the highest-cost model.

This creates a practical optimization problem:

Minimize cost

subject to:

quality >= required threshold
latency <= required threshold
availability >= required threshold

Semantic Routing Can Be Wrong

A semantic router is itself another source of failure.

Suppose the request is:

Explain why this distributed transaction implementation is failing.

The router might classify it as a simple programming question.

But the request may actually require:

Code analysis
Distributed systems reasoning
Long context
Multi-step diagnosis

If the router selects a lightweight model, the final response may be poor.

Therefore, benchmark the router itself.

Router Accuracy

A useful test set might contain:

Simple questions
Coding tasks
Debugging tasks
Architecture questions
Long-context requests
Reasoning-heavy problems
Structured-output requests

For each request:

Expected Route
Actual Route
Correct?

Then calculate:

Routing Accuracy =
Correct Routes / Total Requests

The benchmark should also evaluate the quality of incorrectly routed requests.

A router that achieves high classification accuracy but occasionally makes extremely expensive mistakes may still be unsuitable for production.

Avoiding Routing Loops

A poorly designed failover system can accidentally create loops.

For example:

Model A
  |
  v
Model B
  |
  v
Model A
  |
  v
Model B

Always define a finite fallback chain:

Primary
   |
   v
Fallback 1
   |
   v
Fallback 2
   |
   v
Controlled Failure

Set a maximum number of attempts.

Retry vs Failover

Retrying and failing over are also different.

Retry:
Same model
     |
     v
Try again

Failover:

Model A
   |
   X
   |
Model B

A temporary network failure may justify retrying the same provider.

A provider outage should probably move to another provider.

The decision should be based on failure classification.

Observability

Routing decisions should be visible in telemetry.

A useful log entry might contain:

RequestId
SelectedRoute
Model
Provider
RoutingReason
Latency
RetryCount
FallbackTriggered
FallbackModel
InputTokens
OutputTokens
Success

For example:

RequestId: 84d2
Route: Coding
Primary: ProviderA
Fallback: ProviderB
FallbackTriggered: true
LatencyMs: 2410
Success: true

This makes production incidents significantly easier to investigate.

Common Mistakes

Treating Routing as a Static Rule

A rule such as:

if prompt.Length > 1000

is not a complete semantic router.

Prompt length is only one signal.

Using Failover for Every Error

Not every error is recoverable.

Invalid requests should not automatically be sent to another provider.

Ignoring Cost

A fallback strategy that always chooses the most expensive model can preserve availability while destroying cost efficiency.

Measuring Only Availability

A system can have excellent availability but poor quality or excessive latency.

Not Testing the Router

The routing layer needs its own evaluation dataset.

Allowing Unlimited Retries

Unlimited retry/failover behavior can increase latency, cost, and provider load.

Best Practices

Separate Selection From Recovery

Keep these responsibilities distinct:

Semantic Routing
       |
       v
Model Selection
       |
       v
Execution
       |
       v
Failure Handling
       |
       v
Failover

Use Explicit Policies

Define:

Which models handle which requests
Which failures trigger retries
Which failures trigger failover
Maximum attempts
Maximum latency
Cost limits

Benchmark With Real Workloads

Synthetic prompts are useful, but production-like requests provide better routing signals.

Track Quality and Cost Together

A routing strategy should optimize the entire operating point rather than a single metric.

Add Circuit Breaking for Persistent Provider Failures

If a provider repeatedly fails, temporarily stop sending traffic to it rather than discovering the outage independently on every request.

Keep Fallback Chains Short

A small number of well-tested alternatives is usually easier to operate than a large routing graph.

Advantages and Disadvantages

Semantic Routing Advantages

Can reduce model cost.
Can improve latency.
Matches models to workload characteristics.
Enables workload-specific optimization.
Makes multi-model architectures practical.

Semantic Routing Disadvantages

Router classification can be wrong.
Adds routing latency.
Requires evaluation.
More complex than a single model.
Routing logic requires maintenance.

Failover Routing Advantages

Improves availability.
Protects against provider outages.
Provides resilience during rate limiting.
Can reduce user-visible failures.

Failover Routing Disadvantages

Recovery increases latency.
May increase cost.
Incorrect failure classification can produce unnecessary fallbacks.
Multiple providers increase operational complexity.

Final Thoughts

Semantic routing and failover routing should not be viewed as competing approaches. They address different layers of a production AI system.

Semantic routing answers: "Which model is appropriate for this request?"

Failover routing answers: "What should happen when that model cannot serve the request?"

Microsoft.Extensions.AI provides the abstraction layer needed to build provider-independent AI applications, while routing and resilience policies can be composed around those AI clients.

The strongest architecture combines both approaches: classify the workload, select an appropriate model, execute the request, detect retryable failures, and move to a controlled fallback when necessary.

The real benchmark is not simply whether the system produces an answer. A production routing strategy should demonstrate that it can maintain an acceptable balance between quality, cost, latency, and availability under both normal traffic and failure conditions.