AI applications increasingly depend on external model providers. That dependency creates a reliability problem that is easy to overlook during development.

An application may work perfectly when its preferred model provider is available. But what happens when the provider experiences an outage, returns repeated errors, becomes temporarily unreachable, or cannot handle the current request?

For a production AI application, the answer should not simply be "try again."

A more resilient design can route requests to another compatible provider or model when the preferred provider cannot complete the request.

Microsoft.Extensions.AI now provides routing and failover components designed for this type of scenario. The Microsoft.Extensions.AI ecosystem includes abstractions such as IChatClient, while the newer routing capabilities provide mechanisms for semantic routing and failover between chat clients. Microsoft describes FailoverChatClient as a way to automatically move to another configured chat client when the current client fails.

This changes the reliability architecture of an AI application from:

Application
    |
    v
AI Provider
    |
    v
Response

to:

                    ┌── Provider A
                    │
Application ──> Router
                    │
                    ├── Provider B
                    │
                    └── Provider C

The important engineering question is not whether failover exists. It is how well it behaves under realistic failures and whether the additional complexity is justified.

What Is Microsoft.Extensions.AI?

Microsoft.Extensions.AI provides common abstractions for working with AI services in .NET applications.

Instead of coupling application code directly to a specific provider SDK, developers can work against abstractions such as IChatClient.

A simplified architecture looks like this:

public interface IChatClient
{
    // Conceptual representation of a chat client abstraction.
}

In a real application, the concrete implementation might communicate with a hosted model provider.

The advantage is architectural flexibility.

Your application can depend on an abstraction while the underlying implementation changes.

ASP.NET Core Application
          |
          v
    IChatClient
          |
    ┌─────┴─────┐
    │           │
Provider A   Provider B

This abstraction becomes especially useful when an application needs to switch providers without rewriting the application layer.

Why AI Failover Matters

Traditional web applications have long used redundancy.

For example:

Web Server A
Web Server B
Database Replica
Cache Replica

AI applications have a similar dependency problem, but the failure modes can be different.

An AI provider can experience:

A resilient application should decide which of these conditions justify a retry and which should trigger failover.

That distinction matters.

A retry against the same unavailable service does not provide redundancy.

Failover does.

Understanding FailoverChatClient

The failover approach wraps multiple chat clients and attempts to use another client when a request fails.

Conceptually:

FailoverChatClient
       |
       +---- Primary Chat Client
       |
       +---- Secondary Chat Client
       |
       +---- Tertiary Chat Client

A request might follow this path:

Request
  |
  v
Provider A
  |
  X Failure
  |
  v
Provider B
  |
  v
Response

The application therefore does not need to manually implement provider selection throughout every business service.

That separation is valuable because resilience becomes part of the AI infrastructure layer instead of being duplicated throughout application code.

A Simplified .NET Configuration

The exact APIs should be checked against the version of Microsoft.Extensions.AI used by the application, but the conceptual setup looks like this:

using Microsoft.Extensions.AI;

var builder = WebApplication.CreateBuilder(args);

var primaryClient = new SomeProviderChatClient(
    builder.Configuration["AI:PrimaryApiKey"]!);

var secondaryClient = new AnotherProviderChatClient(
    builder.Configuration["AI:SecondaryApiKey"]!);

var failoverClient = new FailoverChatClient(
    primaryClient,
    secondaryClient);

builder.Services.AddSingleton<IChatClient>(failoverClient);

The important architectural idea is that application services receive IChatClient rather than directly depending on either provider.

For example:

public sealed class SupportAssistant
{
    private readonly IChatClient _chatClient;

    public SupportAssistant(IChatClient chatClient)
    {
        _chatClient = chatClient;
    }

    public async Task<string> AnswerAsync(
        string question,
        CancellationToken cancellationToken)
    {
        // Invoke the configured chat client.
        // Failover is handled below the application layer.
        return await GenerateAnswerAsync(
            question,
            cancellationToken);
    }

    private Task<string> GenerateAnswerAsync(
        string question,
        CancellationToken cancellationToken)
    {
        // Provider-specific implementation omitted.
        throw new NotImplementedException();
    }
}

The application does not need to know which provider ultimately handled the request.

Failover vs Retry

These concepts are related but not identical.

StrategyWhat HappensMain Purpose
RetryRepeat request against same dependencyRecover from transient failure
FailoverMove request to another dependencyRecover from provider failure
Load balancingDistribute requests across providersCapacity and performance
Semantic routingSelect provider based on request characteristicsMatch workload to model

A robust system may combine several of these.

For example:

Request
  |
  v
Primary Provider
  |
  +-- Temporary failure --> Retry
  |
  +-- Persistent failure --> Failover
                              |
                              v
                         Secondary Provider

The exact policy depends on the application's reliability requirements.

Building a Failover Benchmark

The most useful way to evaluate failover is to simulate failures instead of waiting for a real outage.

Create a controlled test environment with multiple fake or test chat clients.

For example:

public sealed class TestChatClient : IChatClient
{
    private readonly bool _shouldFail;

    public TestChatClient(bool shouldFail)
    {
        _shouldFail = shouldFail;
    }

    public async Task<string> CompleteAsync(
        string prompt,
        CancellationToken cancellationToken)
    {
        if (_shouldFail)
        {
            throw new HttpRequestException(
                "Simulated provider failure.");
        }

        return "Successful response";
    }
}

The exact interface implementation will depend on the version of Microsoft.Extensions.AI being tested.

The test should then measure:

Primary Available
Primary Fails
Primary Times Out
Secondary Available
Secondary Fails
All Providers Fail

Metrics Worth Measuring

A failover benchmark should measure more than whether the final request succeeded.

MetricWhy It Matters
Success rateMeasures reliability
Failover rateShows dependency health
Failover latencyMeasures recovery overhead
Total request latencyMeasures user impact
Retry countShows recovery behavior
Provider error rateIdentifies unstable dependencies
Token/AI costMeasures financial impact
Final response qualityEnsures fallback quality remains acceptable

For example:

scenario,primary,secondary,result,latency_ms,retries
normal,healthy,healthy,success,420,0
primary_failure,failed,healthy,success,890,1
secondary_failure,failed,failed,failure,1500,1

These values are illustrative and should not be interpreted as Microsoft or production benchmarks.

Measuring Recovery Latency

One of the most important metrics is the time between primary failure and successful recovery.

A simple benchmark can measure the complete request:

var stopwatch = Stopwatch.StartNew();

try
{
    var response = await chatClient.GetResponseAsync(
        prompt,
        cancellationToken);

    stopwatch.Stop();

    Console.WriteLine(
        $"Request completed in {stopwatch.ElapsedMilliseconds} ms");
}
catch (Exception ex)
{
    stopwatch.Stop();

    Console.WriteLine(
        $"Request failed after {stopwatch.ElapsedMilliseconds} ms");

    throw;
}

For production measurements, use structured telemetry rather than console output.

The benchmark should separately capture:

Primary request latency
        +
Failure detection time
        +
Failover selection time
        +
Secondary request latency
        =
Total recovery latency

This helps identify where the resilience mechanism is adding overhead.

Testing Provider Outages

A useful test suite should simulate several failure types.

Immediate Failure

The primary provider immediately returns an error.

Primary
   |
   X 500
   |
   v
Secondary
   |
   v
Success

This tests basic failover.

Timeout

The primary provider does not respond within the configured timeout.

Primary
   |
   |........
   |
   X Timeout
   |
   v
Secondary

This is particularly important because waiting too long before failing over can make an application appear unavailable even though another provider is healthy.

Rate Limiting

A provider may return a rate-limit response rather than being completely unavailable.

This requires careful policy design.

Automatically switching providers may help, but blindly retrying can make the situation worse.

Authentication Failure

An expired or invalid credential can produce repeated failures.

This should generally not be treated like a normal transient network failure.

Failing over may keep the application operational, but the credential problem should still be surfaced through monitoring.

Failover and Model Compatibility

Switching providers can introduce an important application-level problem: the fallback model may not behave identically to the primary model.

Differences can include:

Therefore, provider failover should not be evaluated only as:

Provider A failed
Provider B responded

The better question is:

Provider A failed
        |
        v
Provider B responded
        |
        v
Did the application still receive a usable result?

For structured workflows, validate the fallback response before returning it to downstream systems.

Failover vs Semantic Routing

Microsoft's routing capabilities also include semantic routing, which solves a different problem.

Failover primarily answers:

"What should I do when my selected provider fails?"

Semantic routing answers:

"Which model should handle this request?"

A simplified architecture is:

                    ┌── Model A
                    │
Request ──> Router ─┼── Model B
                    │
                    └── Model C

The selection may depend on characteristics of the request.

For example:

Simple classification
        |
        v
Lower-cost model


Complex reasoning
        |
        v
More capable model

Failover and semantic routing can therefore complement each other.

A Combined Architecture

A more sophisticated application might use:

                         Incoming Request
                                |
                                v
                       Semantic Router
                         /           \
                        /             \
                       v               v
                 Model A            Model B
                    |                  |
              Failover A          Failover B
                /     \              /     \
               v       v            v       v
          Provider A Provider C Provider B Provider D

This provides both workload-aware routing and redundancy.

However, complexity increases quickly.

Each additional routing layer creates more behavior that needs to be observed and tested.

Common Mistakes

Treating Every Error as Failover-Eligible

Not every failure should trigger a provider switch.

Invalid input, authorization failures, and application bugs may require different handling.

Ignoring Fallback Quality

A successful HTTP response is not necessarily a successful AI response.

Validate the actual application result.

Adding Too Many Providers

Three providers do not automatically provide three times the reliability.

More integrations also mean more credentials, monitoring, testing, and operational complexity.

Failing Over Without Observability

If the application silently switches providers, the team may not realize that the primary provider is unhealthy.

Log and measure failover events.

Ignoring Cost Differences

Fallback providers may have different pricing models.

Reliability engineering and FinOps should be considered together.

Troubleshooting Failover

When failover does not behave as expected, check the following:

  1. Confirm that multiple chat clients are correctly registered.

  2. Verify the fallback client is actually part of the configured pipeline.

  3. Simulate a controlled primary failure.

  4. Confirm that the failure is eligible for failover.

  5. Measure timeout values.

  6. Check logs for provider-specific errors.

  7. Verify fallback authentication.

  8. Compare primary and fallback model capabilities.

  9. Validate the final response.

  10. Confirm that monitoring records the failover event.

A particularly useful test is to intentionally disable the primary provider in a non-production environment and verify that the application continues operating.

Advantages

Disadvantages

Best Practices

Define Failure Policies Explicitly

Document which failures should trigger:

Retry
Failover
Immediate Failure

Do not rely on accidental behavior.

Use Timeouts

A fallback that starts only after an excessively long timeout is not useful for latency-sensitive applications.

Monitor Provider Health

Track:

Provider availability
Error rate
Latency
Rate-limit events
Failover frequency
Fallback success rate

Test the Fallback Path

A fallback path that has never been exercised is not a reliable fallback.

Regularly test it in controlled environments.

Validate Cross-Provider Compatibility

If your application expects structured output, tool calling, or specific model capabilities, verify that the fallback provider supports the required behavior.

Keep Business Logic Provider-Agnostic

Prefer:

Application
    |
    v
IChatClient
    |
    v
Routing / Failover
    |
    v
Provider

over embedding provider-specific logic in every service.

Final Thoughts

AI provider reliability is becoming an application architecture concern rather than simply an SDK configuration detail.

Microsoft.Extensions.AI provides abstractions that make it easier for .NET applications to separate AI application logic from provider-specific implementations. Its routing and failover capabilities extend that architecture by allowing applications to recover from provider failures and, where appropriate, route requests between different AI services.

But failover should not be measured by a single question—"Did the request eventually succeed?"

A production benchmark should measure recovery latency, success rate, fallback quality, provider errors, retry behavior, and cost.

The strongest architecture is not necessarily the one with the largest number of providers. It is the one where failure behavior is predictable, observable, tested, and appropriate for the application's reliability requirements.

For .NET teams building production AI applications, that makes Microsoft.Extensions.AI routing and failover worth evaluating as part of the broader resilience architecture—not merely as another AI API feature.