AI applications increasingly depend on multiple models and providers. A production application may use one model for general conversations, another for complex reasoning, and a third as a backup when the primary service becomes unavailable.
This creates an important reliability question:
What should happen when the preferred AI model fails?
A simple application may return an error immediately:
User Request
|
v
Primary Model
|
X
Error
|
v
500 / Failure
A resilient application can instead use a controlled fallback chain:
User Request
|
v
Primary Model
|
X
Failure
|
v
Secondary Model
|
X
Failure
|
v
Tertiary Model
|
v
Response
Fallbacks can improve availability, but they also introduce additional latency, cost, and complexity.
A well-designed fallback strategy therefore needs clear failure classification, model compatibility, timeout policies, observability, and safeguards against retry storms.
This article explains how to design and implement AI fallback chains in .NET applications and how to evaluate them under realistic failure conditions.
Introduction
Consider an application that normally uses a primary model:
Application
|
v
Primary AI Model
Most requests work normally.
However, production systems can encounter:
Rate limits
Service outages
Network failures
Timeouts
Authentication problems
Invalid responses
Provider-specific errors
Temporary capacity problems
If the application has no fallback strategy, even a temporary failure can become a visible outage.
A fallback chain provides another execution path:
Primary
|
+-- Success --> Response
|
+-- Retryable Failure
|
v
Secondary
|
+-- Success --> Response
|
+-- Failure
|
v
Tertiary
The key is that not every failure should trigger a fallback.
For example, a malformed request caused by application code should generally not be sent repeatedly to different models.
What Is an AI Fallback Chain?
An AI fallback chain is an ordered collection of model or provider options that the application can use when an earlier option cannot successfully complete a request.
For example:
Priority 1: Primary Model
Priority 2: Secondary Model
Priority 3: Emergency Model
A fallback chain can operate across:
Same Provider
|
+--> Model A
+--> Model B
Multiple Providers
|
+--> Provider A
+--> Provider B
+--> Provider C
The second approach can provide stronger provider-level resilience, but it also creates additional integration and governance requirements.
Fallback Is Not the Same as Retry
Retry and fallback solve different problems.
Retry
Retry attempts the same operation again.
Model A
|
X
|
Retry Model A
Fallback
Fallback changes the execution target.
Model A
|
X
|
Model B
A resilient system may use both:
Primary Model
|
X
|
Retry
|
X
|
Secondary Model
However, combining retries and fallbacks without limits can create excessive latency and cost.
Classify Failures Before Falling Back
The most important design decision is determining which failures are eligible for fallback.
A useful classification is:
| Failure | Usually Retryable | Usually Fallbackable |
|---|---|---|
| Timeout | Yes | Yes |
| Temporary network failure | Yes | Yes |
| Rate limit | Sometimes | Yes |
| Service unavailable | Yes | Yes |
| Invalid request | No | Usually No |
| Authentication failure | No | Depends |
| Unsupported operation | No | Depends |
| Content policy rejection | Usually No | Usually No |
| Invalid application input | No | No |
| Malformed model response | Sometimes | Sometimes |
The exact policy depends on the provider and application.
The important principle is:
Do not treat every exception as a reason to switch models.
Define a Failure Classification
A .NET application can represent failure categories explicitly.
public enum AiFailureKind
{
None,
Timeout,
RateLimit,
ServiceUnavailable,
NetworkFailure,
AuthenticationFailure,
InvalidRequest,
UnsupportedOperation,
InvalidResponse,
Unknown
}
Then define a classifier:
public interface IAiFailureClassifier
{
AiFailureKind Classify(Exception exception);
}
This keeps provider-specific exception handling out of the fallback engine.
Create a Fallback Policy
A fallback policy can determine whether a failure should move to the next model.
public interface IFallbackPolicy
{
bool ShouldFallback(
AiFailureKind failure);
}
A simple implementation might be:
public sealed class DefaultFallbackPolicy
: IFallbackPolicy
{
public bool ShouldFallback(
AiFailureKind failure)
{
return failure is
AiFailureKind.Timeout or
AiFailureKind.RateLimit or
AiFailureKind.ServiceUnavailable or
AiFailureKind.NetworkFailure;
}
}
This is safer than:
catch (Exception)
{
// Try another model
}
because unexpected programming errors should not automatically trigger another expensive model request.
Define the Fallback Target
A fallback target should contain enough information to execute the alternative request.
public sealed record AiEndpoint(
string Name,
IChatClient Client,
int Priority);
The chain can then be ordered:
var endpoints = new[]
{
primary,
secondary,
tertiary
}
.OrderBy(x => x.Priority)
.ToArray();
Basic Fallback Executor
A simplified fallback executor might look like this:
public async Task<ChatResponse> ExecuteAsync(
string prompt,
CancellationToken cancellationToken)
{
Exception? lastException = null;
foreach (var endpoint in _endpoints)
{
try
{
return await endpoint.Client.GetResponseAsync(
prompt,
cancellationToken: cancellationToken);
}
catch (Exception ex)
{
lastException = ex;
var failure =
_classifier.Classify(ex);
if (!_policy.ShouldFallback(failure))
{
throw;
}
_logger.LogWarning(
ex,
"AI endpoint {Endpoint} failed with {FailureKind}.",
endpoint.Name,
failure);
}
}
throw new InvalidOperationException(
"All AI endpoints failed.",
lastException);
}
This is intentionally simplified. A production implementation also needs timeout budgets, retry rules, cancellation handling, telemetry, response validation, and provider-specific behavior.
Fallback Chain Architecture
A production architecture can look like:
Application
|
v
AI Client Layer
|
v
Failure Classifier
|
v
Fallback Policy
|
+-----------+-----------+
| | |
v v v
Model A Model B Model C
| | |
+-----------+-----------+
|
v
Final Result
The application does not need to understand every provider-specific failure.
The fallback layer handles that responsibility.
Preserve the Original Request
A fallback should generally receive the same logical request.
For example:
User Request
|
+--> Primary
|
+--> Secondary
|
+--> Tertiary
However, the request may need adaptation if the models have different capabilities.
Examples include:
Different context limits
Different tool support
Different structured-output support
Different multimodal capabilities
Different system-prompt requirements
Therefore, a fallback chain should validate compatibility before execution.
Capability Matching
Suppose the primary model supports:
Text
Images
Tool Calling
Structured Output
while the fallback supports only:
Text
The fallback may not be valid for every request.
Define capabilities explicitly:
public sealed record ModelCapabilities(
bool SupportsTools,
bool SupportsImages,
bool SupportsStructuredOutput,
int ContextLimit);
Then evaluate the request before selecting a fallback.
if (request.RequiresTools &&
!endpoint.Capabilities.SupportsTools)
{
continue;
}
This prevents the application from switching to a model that cannot actually complete the task.
Fallback Compatibility
Not every fallback needs to be functionally identical.
A fallback can be:
Equivalent
Designed to provide essentially the same capability.
Reduced Capability
Provides a degraded but acceptable experience.
For example:
Primary:
Full agentic workflow
Fallback:
Answer without tool execution
Emergency
Provides a minimal response when normal AI processing is unavailable.
For example:
Primary:
Advanced model
Secondary:
Standard model
Emergency:
Static knowledge response
The degradation should be intentional rather than accidental.
Timeout Budgets
Fallback chains can become slow if every endpoint receives the full timeout.
Consider:
Primary timeout = 30 seconds
Secondary timeout = 30 seconds
Tertiary timeout = 30 seconds
Worst-case latency could approach:
90 seconds
That may be unacceptable.
Instead, define a total request budget:
Total Budget = 15 seconds
Then allocate time dynamically.
Primary
|
| 8 sec
v
Secondary
|
| 5 sec
v
Emergency
|
| 2 sec
v
Final Result
The exact values depend on the workload.
Cancellation Tokens
Always propagate cancellation.
using var timeout =
CancellationTokenSource.CreateLinkedTokenSource(
cancellationToken);
timeout.CancelAfter(
TimeSpan.FromSeconds(15));
await client.GetResponseAsync(
prompt,
cancellationToken: timeout.Token);
If the user cancels the request, the fallback chain should not continue making unnecessary model calls.
Retry Plus Fallback
A common pattern is:

Join the conversation! Your thoughts help the community grow.