Microsoft.Extensions.AI Routing: Benchmarking Cost, Latency, and Failover

Aarav Patel
1d
172
0
2

Article

Introduction

Modern .NET applications increasingly use more than one AI model. A production application might use a smaller model for simple classification, a more capable model for complex reasoning, and another provider as a fallback when the primary service is unavailable.

That creates a routing problem.

Instead of sending every request to one model, the application needs to decide which IChatClient should handle each request. The decision may depend on task complexity, latency, cost, availability, or application policy.

Microsoft.Extensions.AI provides the IChatClient abstraction for interacting with AI services and supports composable client pipelines through ChatClientBuilder. This makes it possible to build routing, caching, telemetry, rate limiting, and other middleware around a common interface.

Routing sounds straightforward, but production routing is not simply:

If request is simple -> Model A
Otherwise -> Model B

A useful router must also consider what happens when a model becomes slow, unavailable, rate-limited, or unexpectedly expensive.

This article shows how to design and benchmark a practical AI routing layer in .NET.

Why AI Model Routing Matters

Different models can have different characteristics.

For example, an application may have:

Workload	Preferred model characteristic
Classification	Low latency
Simple extraction	Low cost
General chat	Balanced
Complex reasoning	Higher capability
Large-context analysis	Large context support
Fallback	High availability

Using the most capable model for every request can unnecessarily increase cost and latency.

Using the cheapest model for every request can reduce response quality.

Routing attempts to find a better balance.

A simplified architecture looks like this:

                    +----------------+
Request ----------->| AI Router      |
                    +-------+--------+
                            |
              +-------------+-------------+
              |             |             |
              v             v             v
          Model A        Model B       Model C
          Fast           Balanced      Advanced

The router becomes an application-level policy engine.

Understanding IChatClient

IChatClient provides a common abstraction for chat-capable AI services. It supports regular and streaming responses and accepts ChatOptions for per-request configuration.

A basic application can work against the abstraction rather than a provider-specific implementation:

using Microsoft.Extensions.AI;

public sealed class AiService(IChatClient chatClient)
{
    public async Task<string> AskAsync(
        string prompt,
        CancellationToken cancellationToken = default)
    {
        var response = await chatClient.GetResponseAsync(
            prompt,
            cancellationToken: cancellationToken);

        return response.Text;
    }
}

The benefit is separation of application logic from the underlying model provider.

The same abstraction can also be wrapped with additional functionality such as distributed caching, telemetry, rate limiting, and custom delegating clients.

That composability is particularly useful for routing.

A Simple Routing Strategy

Start with an explicit routing policy.

For example:

public enum AiTaskType
{
    Simple,
    General,
    Complex
}

Then map each workload to an AI client:

using Microsoft.Extensions.AI;

public sealed class AiRouter(
    IChatClient fastClient,
    IChatClient generalClient,
    IChatClient advancedClient)
{
    public IChatClient SelectClient(AiTaskType taskType)
    {
        return taskType switch
        {
            AiTaskType.Simple => fastClient,
            AiTaskType.General => generalClient,
            AiTaskType.Complex => advancedClient,
            _ => generalClient
        };
    }
}

This is static routing.

It is easy to understand and easy to test, but it does not react to runtime conditions.

If the advanced model becomes unavailable, the router still selects it.

Production systems usually need a second layer: availability-aware routing.

Adding Failover

Suppose the primary model fails.

The router can attempt a fallback client:

using Microsoft.Extensions.AI;

public sealed class FallbackChatService(
    IChatClient primary,
    IChatClient fallback)
{
    public async Task<ChatResponse> GetResponseAsync(
        IEnumerable<ChatMessage> messages,
        CancellationToken cancellationToken = default)
    {
        try
        {
            return await primary.GetResponseAsync(
                messages,
                cancellationToken: cancellationToken);
        }
        catch (HttpRequestException)
        {
            return await fallback.GetResponseAsync(
                messages,
                cancellationToken: cancellationToken);
        }
    }
}

This demonstrates the basic idea, but production code should not blindly treat every exception as a failover condition.

A validation error should not necessarily cause the request to be sent to another provider.

A useful policy distinguishes failures:

Provider unavailable     -> Failover
Rate limited              -> Retry/failover
Timeout                   -> Failover
Invalid request           -> Return error
Authentication failure   -> Alert/fail safely
Content-policy response  -> Apply application policy

The classification should be specific to the providers and infrastructure used by the application.

Latency-Based Routing

Availability is only one dimension.

A model can be technically available while responding too slowly for the application's SLA.

Suppose two models are available:

Model A
Latency: 250 ms

Model B
Latency: 1,500 ms

If the task is simple and both produce acceptable results, Model A may be preferable.

A simple latency-aware policy can maintain rolling measurements:

public sealed record ModelHealth(
    string Name,
    double AverageLatencyMs,
    double FailureRate);

The router can then rank eligible models.

public static ModelHealth SelectFastest(
    IEnumerable<ModelHealth> models)
{
    return models
        .Where(m => m.FailureRate < 0.05)
        .OrderBy(m => m.AverageLatencyMs)
        .First();
}

The 5% value here is an application policy example, not a universal production threshold.

Thresholds should come from the application's requirements and observed workload.

Cost-Aware Routing

Cost is another routing dimension.

A practical router can assign each request a complexity class:

Simple
  -> Low-cost model

Moderate
  -> Standard model

Complex
  -> Advanced model

For example:

public sealed record RoutingDecision(
    string Model,
    string Reason);

public static RoutingDecision Route(AiTaskType task)
{
    return task switch
    {
        AiTaskType.Simple =>
            new("fast-model", "Simple workload"),

        AiTaskType.General =>
            new("general-model", "Standard workload"),

        AiTaskType.Complex =>
            new("advanced-model", "Complex workload"),

        _ =>
            new("general-model", "Default route")
    };
}

The important design principle is to make routing decisions observable.

If the router selects an expensive model, you should be able to determine why.

A Better Routing Pipeline

Routing should not be the only middleware in the AI client pipeline.

Microsoft.Extensions.AI supports composable client functionality. Microsoft documents examples involving distributed caching, function invocation, OpenTelemetry, logging, rate limiting, and custom delegating clients.

A conceptual pipeline can look like:

Application
    |
    v
Routing
    |
    v
Rate Limiting
    |
    v
Telemetry
    |
    v
Caching
    |
    v
Provider Client

The exact ordering depends on the desired semantics.

For example, if caching occurs before the provider call, a cache hit may avoid the downstream model request entirely.

Microsoft's documentation shows UseDistributedCache() and UseOpenTelemetry() being composed around an IChatClient.

Benchmarking the Router

A routing implementation should be measured rather than judged by intuition.

At minimum, measure:

Metric	Purpose
Request count	Workload volume
Average latency	General responsiveness
P50 latency	Typical request
P95 latency	Slow-tail behavior
P99 latency	Extreme tail
Failure rate	Reliability
Failover rate	Primary-provider health
Cost/request	Economic efficiency
Cache-hit rate	Avoided provider calls
Model distribution	Routing behavior

P50 alone is not enough.

A model might have acceptable average latency while occasionally producing very slow responses.

P95 and P99 help expose those tail conditions.

Building a Simple Benchmark Harness

A small benchmark harness can execute the same workload against different routing policies.

public sealed record BenchmarkResult(
    string Strategy,
    TimeSpan Duration,
    bool Success);

public async Task<BenchmarkResult> RunAsync(
    string strategy,
    Func<Task> operation)
{
    var stopwatch = Stopwatch.StartNew();

    try
    {
        await operation();

        stopwatch.Stop();

        return new BenchmarkResult(
            strategy,
            stopwatch.Elapsed,
            true);
    }
    catch
    {
        stopwatch.Stop();

        return new BenchmarkResult(
            strategy,
            stopwatch.Elapsed,
            false);
    }
}

For meaningful results, run enough representative requests to capture normal and failure conditions.

Do not treat a handful of requests as a production benchmark.

Comparing Static and Dynamic Routing

Consider three strategies.

Strategy A: Static Model

Every request goes to one model.

100% -> Model A

This is simple but provides no workload adaptation.

Strategy B: Task-Based Routing

Requests are classified before selection.

Simple  -> Model A
General -> Model B
Complex -> Model C

This can improve cost and capability alignment.

Strategy C: Health-Aware Routing

The router considers task type plus current model health.

Task Type
    +
Latency
    +
Failure Rate
    +
Availability
    |
    v
Routing Decision

This is more sophisticated but also more complex to operate.

Strategy	Complexity	Adaptability	Failover	Cost Control
Static	Low	Low	Low	Low
Task-based	Medium	Medium	Medium	High
Health-aware	High	High	High	High

The best strategy depends on the application's requirements.

Avoiding Routing Oscillation

Dynamic routing can introduce a subtle problem: oscillation.

Imagine the router switches providers whenever one model becomes slightly slower.

Model A slower
    |
    v
Switch to B

Model B slower
    |
    v
Switch to A

Model A slower
    |
    v
Switch to B

This can produce unstable traffic patterns.

Use mechanisms such as:

Minimum observation windows
Hysteresis
Failure thresholds
Cooldown periods
Rolling averages
Circuit breakers

The goal is to avoid making routing decisions from a single slow request.

Circuit Breakers and Provider Health

A circuit breaker can temporarily stop routing requests to an unhealthy provider.

Conceptually:

Healthy
   |
   | failures exceed threshold
   v
Open
   |
   | cooldown
   v
Half-Open
   |
   | successful probe
   v
Healthy

This prevents repeatedly sending requests to a provider that is already failing.

The same principle can be applied at the model or endpoint level.

Observability Is Part of Routing

A routing decision without telemetry is difficult to troubleshoot.

Record information such as:

Request ID
Task type
Selected model
Fallback model
Latency
Failure classification
Retry count
Token usage, where available
Routing reason

Microsoft.Extensions.AI provides an OpenTelemetryChatClient that can add tracing and metrics around IChatClient operations.

A useful log entry might conceptually look like:

{
  "taskType": "Complex",
  "selectedModel": "advanced-model",
  "fallbackUsed": false,
  "latencyMs": 820,
  "routingReason": "Complexity score"
}

Avoid logging sensitive prompts or responses unless your data-handling policy explicitly permits it.

Common Mistakes

Using the Most Powerful Model for Everything

This can increase cost without improving every workload.

Failing Over on Every Exception

Some errors are caused by invalid application input and will fail against every provider.

Routing Based on One Request

Individual requests are noisy. Use aggregated health signals.

Ignoring Tail Latency

Average latency can hide problematic P95 or P99 behavior.

Building a Router Without Observability

If developers cannot determine why a model was selected, troubleshooting becomes difficult.

Making Routing Too Complex Too Early

Start with explicit policies and measurable criteria. Add dynamic behavior only when the workload justifies it.

Best Practices

Start with explicit routing rules. Keep the first implementation understandable.
Separate routing from provider-specific code. Depend on IChatClient where practical.
Measure latency by percentile. Track P50, P95, and P99.
Track failure rates per provider. Availability should influence routing.
Classify failures before failover. Not every exception should trigger another model request.
Use bounded retries and cooldowns. Prevent request amplification and routing oscillation.
Make routing decisions observable. Record why a model was selected.
Evaluate cost and quality together. A cheaper model is not automatically a better choice.
Use caching where appropriate. Microsoft.Extensions.AI provides caching abstractions that can be composed into an IChatClient pipeline.
Benchmark with representative workloads. Routing behavior depends heavily on request mix and provider characteristics.

Frequently Asked Questions

What is Microsoft.Extensions.AI?

Microsoft.Extensions.AI provides .NET abstractions and components for working with AI services. Its IChatClient abstraction allows applications to interact with different chat-capable implementations through a common programming model.

Does IChatClient automatically route between models?

No. IChatClient provides the abstraction; routing policy is an application or library concern.

Should routing be based on cost or latency?

Usually both, along with quality and availability. A production router should optimize for the application's actual requirements rather than one metric.

Is failover the same as routing?

No. Routing determines where a request should go. Failover determines what to do when the selected path cannot successfully handle the request.

Can I add telemetry without rewriting my AI application?

Microsoft.Extensions.AI supports composable client pipelines, including OpenTelemetry-based instrumentation. This allows cross-cutting functionality to be layered around an IChatClient.

Should every AI application use multiple models?

No. Multiple models introduce operational and testing complexity. A single model may be the right choice when its quality, cost, latency, and availability satisfy the application's requirements.

Conclusion

AI model routing is becoming an important architectural concern as applications use multiple models and providers. The challenge is not simply choosing a model. It is deciding when each model should be used and what should happen when the preferred model is slow, unavailable, rate-limited, or unsuitable for the workload.

Microsoft.Extensions.AI provides a useful foundation through IChatClient and composable client pipelines.

A practical routing architecture should begin with explicit policies, then add health-aware failover, latency measurement, cost controls, caching, and observability as needed. Most importantly, benchmark the complete system instead of assuming that a routing strategy is better because it looks more sophisticated.

The goal is not to route every request to the cheapest or fastest model. The goal is to consistently select a model that provides the required quality and reliability at an acceptable latency and cost.