AI applications increasingly depend on external model providers. That dependency creates a reliability problem that is easy to overlook during development.
An application may work perfectly when its preferred model provider is available. But what happens when the provider experiences an outage, returns repeated errors, becomes temporarily unreachable, or cannot handle the current request?
For a production AI application, the answer should not simply be "try again."
A more resilient design can route requests to another compatible provider or model when the preferred provider cannot complete the request.
Microsoft.Extensions.AI now provides routing and failover components designed for this type of scenario. The Microsoft.Extensions.AI ecosystem includes abstractions such as IChatClient, while the newer routing capabilities provide mechanisms for semantic routing and failover between chat clients. Microsoft describes FailoverChatClient as a way to automatically move to another configured chat client when the current client fails.
This changes the reliability architecture of an AI application from:
Application
|
v
AI Provider
|
v
Response
to:
┌── Provider A
│
Application ──> Router
│
├── Provider B
│
└── Provider C
The important engineering question is not whether failover exists. It is how well it behaves under realistic failures and whether the additional complexity is justified.
What Is Microsoft.Extensions.AI?
Microsoft.Extensions.AI provides common abstractions for working with AI services in .NET applications.
Instead of coupling application code directly to a specific provider SDK, developers can work against abstractions such as IChatClient.
A simplified architecture looks like this:
public interface IChatClient
{
// Conceptual representation of a chat client abstraction.
}
In a real application, the concrete implementation might communicate with a hosted model provider.
The advantage is architectural flexibility.
Your application can depend on an abstraction while the underlying implementation changes.
ASP.NET Core Application
|
v
IChatClient
|
┌─────┴─────┐
│ │
Provider A Provider B
This abstraction becomes especially useful when an application needs to switch providers without rewriting the application layer.
Why AI Failover Matters
Traditional web applications have long used redundancy.
For example:
Web Server A
Web Server B
Database Replica
Cache Replica
AI applications have a similar dependency problem, but the failure modes can be different.
An AI provider can experience:
A resilient application should decide which of these conditions justify a retry and which should trigger failover.
That distinction matters.
A retry against the same unavailable service does not provide redundancy.
Failover does.
Understanding FailoverChatClient
The failover approach wraps multiple chat clients and attempts to use another client when a request fails.
Conceptually:
FailoverChatClient
|
+---- Primary Chat Client
|
+---- Secondary Chat Client
|
+---- Tertiary Chat Client
A request might follow this path:
Request
|
v
Provider A
|
X Failure
|
v
Provider B
|
v
Response
The application therefore does not need to manually implement provider selection throughout every business service.
That separation is valuable because resilience becomes part of the AI infrastructure layer instead of being duplicated throughout application code.
A Simplified .NET Configuration
The exact APIs should be checked against the version of Microsoft.Extensions.AI used by the application, but the conceptual setup looks like this:
using Microsoft.Extensions.AI;
var builder = WebApplication.CreateBuilder(args);
var primaryClient = new SomeProviderChatClient(
builder.Configuration["AI:PrimaryApiKey"]!);
var secondaryClient = new AnotherProviderChatClient(
builder.Configuration["AI:SecondaryApiKey"]!);
var failoverClient = new FailoverChatClient(
primaryClient,
secondaryClient);
builder.Services.AddSingleton<IChatClient>(failoverClient);
The important architectural idea is that application services receive IChatClient rather than directly depending on either provider.
For example:
public sealed class SupportAssistant
{
private readonly IChatClient _chatClient;
public SupportAssistant(IChatClient chatClient)
{
_chatClient = chatClient;
}
public async Task<string> AnswerAsync(
string question,
CancellationToken cancellationToken)
{
// Invoke the configured chat client.
// Failover is handled below the application layer.
return await GenerateAnswerAsync(
question,
cancellationToken);
}
private Task<string> GenerateAnswerAsync(
string question,
CancellationToken cancellationToken)
{
// Provider-specific implementation omitted.
throw new NotImplementedException();
}
}
The application does not need to know which provider ultimately handled the request.
Failover vs Retry
These concepts are related but not identical.
| Strategy | What Happens | Main Purpose |
|---|
| Retry | Repeat request against same dependency | Recover from transient failure |
| Failover | Move request to another dependency | Recover from provider failure |
| Load balancing | Distribute requests across providers | Capacity and performance |
| Semantic routing | Select provider based on request characteristics | Match workload to model |
A robust system may combine several of these.
For example:
Request
|
v
Primary Provider
|
+-- Temporary failure --> Retry
|
+-- Persistent failure --> Failover
|
v
Secondary Provider
The exact policy depends on the application's reliability requirements.
Building a Failover Benchmark
The most useful way to evaluate failover is to simulate failures instead of waiting for a real outage.
Create a controlled test environment with multiple fake or test chat clients.
For example:
public sealed class TestChatClient : IChatClient
{
private readonly bool _shouldFail;
public TestChatClient(bool shouldFail)
{
_shouldFail = shouldFail;
}
public async Task<string> CompleteAsync(
string prompt,
CancellationToken cancellationToken)
{
if (_shouldFail)
{
throw new HttpRequestException(
"Simulated provider failure.");
}
return "Successful response";
}
}
The exact interface implementation will depend on the version of Microsoft.Extensions.AI being tested.
The test should then measure:
Primary Available
Primary Fails
Primary Times Out
Secondary Available
Secondary Fails
All Providers Fail
Metrics Worth Measuring
A failover benchmark should measure more than whether the final request succeeded.
| Metric | Why It Matters |
|---|
| Success rate | Measures reliability |
| Failover rate | Shows dependency health |
| Failover latency | Measures recovery overhead |
| Total request latency | Measures user impact |
| Retry count | Shows recovery behavior |
| Provider error rate | Identifies unstable dependencies |
| Token/AI cost | Measures financial impact |
| Final response quality | Ensures fallback quality remains acceptable |
For example:
scenario,primary,secondary,result,latency_ms,retries
normal,healthy,healthy,success,420,0
primary_failure,failed,healthy,success,890,1
secondary_failure,failed,failed,failure,1500,1
These values are illustrative and should not be interpreted as Microsoft or production benchmarks.
Measuring Recovery Latency
One of the most important metrics is the time between primary failure and successful recovery.
A simple benchmark can measure the complete request:
var stopwatch = Stopwatch.StartNew();
try
{
var response = await chatClient.GetResponseAsync(
prompt,
cancellationToken);
stopwatch.Stop();
Console.WriteLine(
$"Request completed in {stopwatch.ElapsedMilliseconds} ms");
}
catch (Exception ex)
{
stopwatch.Stop();
Console.WriteLine(
$"Request failed after {stopwatch.ElapsedMilliseconds} ms");
throw;
}
For production measurements, use structured telemetry rather than console output.
The benchmark should separately capture:
Primary request latency
+
Failure detection time
+
Failover selection time
+
Secondary request latency
=
Total recovery latency
This helps identify where the resilience mechanism is adding overhead.
Testing Provider Outages
A useful test suite should simulate several failure types.
Immediate Failure
The primary provider immediately returns an error.
Primary
|
X 500
|
v
Secondary
|
v
Success
This tests basic failover.
Timeout
The primary provider does not respond within the configured timeout.
Primary
|
|........
|
X Timeout
|
v
Secondary
This is particularly important because waiting too long before failing over can make an application appear unavailable even though another provider is healthy.
Rate Limiting
A provider may return a rate-limit response rather than being completely unavailable.
This requires careful policy design.
Automatically switching providers may help, but blindly retrying can make the situation worse.
Authentication Failure
An expired or invalid credential can produce repeated failures.
This should generally not be treated like a normal transient network failure.
Failing over may keep the application operational, but the credential problem should still be surfaced through monitoring.
Failover and Model Compatibility
Switching providers can introduce an important application-level problem: the fallback model may not behave identically to the primary model.
Differences can include:
Therefore, provider failover should not be evaluated only as:
Provider A failed
Provider B responded
The better question is:
Provider A failed
|
v
Provider B responded
|
v
Did the application still receive a usable result?
For structured workflows, validate the fallback response before returning it to downstream systems.
Failover vs Semantic Routing
Microsoft's routing capabilities also include semantic routing, which solves a different problem.
Failover primarily answers:
"What should I do when my selected provider fails?"
Semantic routing answers:
"Which model should handle this request?"
A simplified architecture is:
┌── Model A
│
Request ──> Router ─┼── Model B
│
└── Model C
The selection may depend on characteristics of the request.
For example:
Simple classification
|
v
Lower-cost model
Complex reasoning
|
v
More capable model
Failover and semantic routing can therefore complement each other.
A Combined Architecture
A more sophisticated application might use:
Incoming Request
|
v
Semantic Router
/ \
/ \
v v
Model A Model B
| |
Failover A Failover B
/ \ / \
v v v v
Provider A Provider C Provider B Provider D
This provides both workload-aware routing and redundancy.
However, complexity increases quickly.
Each additional routing layer creates more behavior that needs to be observed and tested.
Common Mistakes
Treating Every Error as Failover-Eligible
Not every failure should trigger a provider switch.
Invalid input, authorization failures, and application bugs may require different handling.
Ignoring Fallback Quality
A successful HTTP response is not necessarily a successful AI response.
Validate the actual application result.
Adding Too Many Providers
Three providers do not automatically provide three times the reliability.
More integrations also mean more credentials, monitoring, testing, and operational complexity.
Failing Over Without Observability
If the application silently switches providers, the team may not realize that the primary provider is unhealthy.
Log and measure failover events.
Ignoring Cost Differences
Fallback providers may have different pricing models.
Reliability engineering and FinOps should be considered together.
Troubleshooting Failover
When failover does not behave as expected, check the following:
Confirm that multiple chat clients are correctly registered.
Verify the fallback client is actually part of the configured pipeline.
Simulate a controlled primary failure.
Confirm that the failure is eligible for failover.
Measure timeout values.
Check logs for provider-specific errors.
Verify fallback authentication.
Compare primary and fallback model capabilities.
Validate the final response.
Confirm that monitoring records the failover event.
A particularly useful test is to intentionally disable the primary provider in a non-production environment and verify that the application continues operating.
Advantages
Reduces dependence on a single AI provider.
Can improve application availability.
Centralizes provider resilience.
Works naturally with .NET dependency injection and abstractions.
Makes provider switching less invasive to application code.
Can be combined with routing strategies.
Disadvantages
Adds configuration and operational complexity.
Fallback models may produce different results.
Additional providers introduce additional credentials and integrations.
Failover can increase latency.
Costs can become harder to predict.
Incorrect failure policies can cause unnecessary provider switching.
Best Practices
Define Failure Policies Explicitly
Document which failures should trigger:
Retry
Failover
Immediate Failure
Do not rely on accidental behavior.
Use Timeouts
A fallback that starts only after an excessively long timeout is not useful for latency-sensitive applications.
Monitor Provider Health
Track:
Provider availability
Error rate
Latency
Rate-limit events
Failover frequency
Fallback success rate
Test the Fallback Path
A fallback path that has never been exercised is not a reliable fallback.
Regularly test it in controlled environments.
Validate Cross-Provider Compatibility
If your application expects structured output, tool calling, or specific model capabilities, verify that the fallback provider supports the required behavior.
Keep Business Logic Provider-Agnostic
Prefer:
Application
|
v
IChatClient
|
v
Routing / Failover
|
v
Provider
over embedding provider-specific logic in every service.
Final Thoughts
AI provider reliability is becoming an application architecture concern rather than simply an SDK configuration detail.
Microsoft.Extensions.AI provides abstractions that make it easier for .NET applications to separate AI application logic from provider-specific implementations. Its routing and failover capabilities extend that architecture by allowing applications to recover from provider failures and, where appropriate, route requests between different AI services.
But failover should not be measured by a single question—"Did the request eventually succeed?"
A production benchmark should measure recovery latency, success rate, fallback quality, provider errors, retry behavior, and cost.
The strongest architecture is not necessarily the one with the largest number of providers. It is the one where failure behavior is predictable, observable, tested, and appropriate for the application's reliability requirements.
For .NET teams building production AI applications, that makes Microsoft.Extensions.AI routing and failover worth evaluating as part of the broader resilience architecture—not merely as another AI API feature.
Join the conversation! Your thoughts help the community grow.