Wrapping an AI SDK behind an interface is a reasonable place to start. For the first feature, that can be enough. Then another team adds a summarisation feature. A background process starts classifying messages. Someone introduces a more capable model for difficult requests. Each feature acquires its own timeout, prompt configuration and retry behaviour. Soon, changing the provider is one of the easier problems. Explaining why yesterday’s requests became slower, more expensive or less accurate takes considerably more work. An enterprise AI gateway earns its place by making those decisions explicit and observable. The provider abstraction is one component of that design.
Start with the operation the application needs
A provider interface often begins with something like this:
public interface IAiProvider
{
Task<string> CompleteAsync(
string prompt,
CancellationToken cancellationToken);
}
It makes the SDK replaceable, but the caller still has to decide almost everything: which instructions to send, how much output to request, how long to wait and whether another attempt is acceptable. Consider a support system that generates a short summary of a customer conversation. The application knows it needs a support summary. It shouldn't need to know the deployment name of the model currently approved to produce one.
An operation-specific request gives the gateway something useful to apply policy to:
public sealed record SupportSummaryRequest(string Conversation);
public sealed record AiExecutionMetadata(
string Operation,
string RouteVersion,
string PromptVersion,
string ModelDeployment,
string? ReportedModel,
long? InputTokens,
long? OutputTokens);
public sealed record SupportSummaryResult(
string Summary,
AiExecutionMetadata Metadata);
public interface ISupportSummaryGateway
{
Task<SupportSummaryResult> SummariseAsync(
SupportSummaryRequest request,
CancellationToken cancellationToken);
}
The gateway resolves the route and prompt internally. The caller receives the result alongside metadata identifying how it was produced. This doesn't mean creating a public interface for every prompt. A generic gateway with typed operation definitions can work too. The useful distinction is that callers request a supported capability, and the gateway resolves the execution policy.
Put policy between the caller and the provider
The gateway needs enough context to make a decision without inheriting the entire business workflow. For our summary operation, a route could define an approved deployment, a prompt version, an output limit and a deadline. An authenticated tenant or workload identity can further restrict which routes are eligible.
The exact deployment shape is a separate decision. You can begin with a module inside a modular monolith or a shared library used by a few services. A separate gateway service becomes useful when independent applications need centrally enforced access, quotas or routing changes. A shared library distributes implementation. A central service can enforce policy at runtime. Those have different operational costs, so choose the boundary based on what you need to control.
Route by capability before cost
A cheaper model is only useful when it can complete the operation to the required standard. For support summaries, an eligible route might need to support the expected conversation length and preserve agreed facts reliably. Another operation might require image input, structured output or tool calling. Route selection should first identify eligible deployments, then apply preferences such as cost or latency. Data handling restrictions must also filter candidates before selection; they shouldn't be relaxed during an outage. An explicit operation-to-route mapping is often a good starting point. You already know whether the caller is requesting a summary or a classification. Introducing another model to discover that information adds another dependency to a decision the application can make directly. Keep the mapping versioned. If a route alias changes from one deployment to another, an old request record should still identify the configuration used at the time.
In .NET, Microsoft.Extensions.AI provides IChatClient as a common client abstraction, with middleware support for concerns such as telemetry and caching. That gives you useful infrastructure to build on without writing another SDK wrapper. Microsoft also documents experimental routing and failover clients. Check their package status before adopting them; your own eligibility and quality policies still need to be defined.
Give the whole operation a deadline
A provider timeout limits one part of a request. The user experiences everything: admission, prompt resolution, provider execution, retry delays and any fallback attempt. Suppose a feature has an illustrative ten-second budget. If its first attempt consumes eight seconds, a fallback cannot automatically receive another ten. Every stage needs to respect the time remaining. For a small non-streaming example, a linked cancellation token can propagate the caller’s cancellation alongside a gateway deadline:
using Microsoft.Extensions.AI;
public sealed class SummaryModelExecutor(IChatClient client)
{
public async Task<ChatResponse> ExecuteAsync(
string instructions,
string conversation,
TimeSpan remainingBudget,
CancellationToken cancellationToken)
{
if (remainingBudget <= TimeSpan.Zero)
{
throw new TimeoutException("The AI operation deadline has expired.");
}
using var deadline =
CancellationTokenSource.CreateLinkedTokenSource(cancellationToken);
deadline.CancelAfter(remainingBudget);
ChatMessage[] messages =
[
new(ChatRole.System, instructions),
new(ChatRole.User, conversation)
];
return await client.GetResponseAsync(
messages,
new ChatOptions { MaxOutputTokens = 400 },
deadline.Token);
}
}
This is an illustrative execution component, rather than a complete gateway. Client registration, route resolution, prompt storage, validation and usage accounting sit around it. The configured client must support and honour the requested output limit. The outer operation owns the deadline and passes the remaining budget to this component. Lower-level timeouts and retries must fit inside it. Cancellation is cooperative: cancelling the local wait doesn't prove that remote processing stopped or that the request incurred no charge. Streaming requires additional decisions. Measure time to first content as well as total duration. Once the caller has received part of an answer, silently switching models could splice together two different responses. Make interrupted output an explicit outcome.
Decide who owns retries
Retries can appear in the provider SDK, an HTTP resilience handler, the gateway and the calling application. If several layers retry independently, one logical operation can generate far more provider calls than expected. The latency and usage records become difficult to explain because each layer believes it is handling a small local failure. Pick an owner for the attempt budget and configure the other layers deliberately. Honour provider backoff guidance when available, and stop when the operation has insufficient time remaining. Microsoft’s HTTP resilience documentation describes both total and per-attempt timeouts, and notes that the standard handler retries all HTTP methods by default. Review the actual handler and SDK configuration rather than assuming generation requests receive a single attempt.
The failure category should influence the response:
Outcome | Gateway behaviour |
|---|
Temporary provider failure | Consider a bounded retry within the remaining budget |
Invalid credentials or unsupported request | Fail and expose the configuration problem |
Caller cancellation | Stop further attempts |
Safety refusal | Preserve the refusal outcome; don't route around it |
Invalid or truncated output | Record the reason and follow an explicit repair or failure policy |
An HTTP success doesn't establish that the application has a usable result. A response can be incomplete, fail schema validation or omit information the operation requires. Keep tool execution separately controlled. Retrying a generation request and repeating an external action are different decisions. A model request that can lead to a payment or a record update needs operation-level protection against duplicate effects.
Treat fallback as a quality change
Fallback can improve availability, but another deployment may produce different answers, support different options or handle the same prompt differently. For our summary feature, the fallback should pass an evaluation set covering the same required behaviour as the primary route. That includes preserving dates and amounts, distinguishing customer statements from confirmed facts and handling long conversations. It doesn't need to use identical wording. It needs to satisfy the operation’s acceptance criteria. Both routes must also satisfy the same access and data handling requirements. A provider outage doesn't authorise sending a conversation to an otherwise ineligible service. Record fallback use explicitly. A dashboard that shows a healthy overall success rate can hide a sustained shift to a slower or less accurate route.
Version the prompt and the configuration around it
“The model changed its answer” is difficult to investigate if the application stores only a model alias and the latest prompt text. For the summary operation, retain immutable prompt versions and record which one was used. Keep the route configuration, relevant generation settings and validator version identifiable too. If retrieval supplies context, record enough authorised provenance to identify the retrieval configuration and source versions. Don't assume a prompt version explains changes caused by a different set of retrieved documents. A central prompt store can make rollout easier, but mutable text under a stable name makes historical investigation harder. Resolve a name such as support-summary to a specific version, and retain that version when promoting the next one. These records support investigation and evaluation. They don't guarantee an identical response on replay: model execution can vary, and providers may update the systems behind a deployment.
Count attempts as well as completed operations
One successful summary can conceal a failed primary attempt and a successful fallback. Track both the logical operation and each provider attempt. An operation record explains what the caller experienced: duration, final outcome and whether fallback was used. Attempt records explain which deployment was called, how long it took and what usage was reported. Keep missing token usage nullable. If a connection fails before usage is returned, zero would imply knowledge you don't have. Estimated cost also needs a pricing version and a stated calculation. Different providers can expose different usage categories, so a simple input-plus-output formula may not describe the eventual bill. Reconcile estimates against billing data where possible. Quotas require a separate mechanism. A dashboard showing yesterday’s spend won't prevent concurrent requests from exceeding today's budget. Where hard admission limits are needed, reserve a conservative amount atomically before execution and reconcile afterwards, with an expiry policy for abandoned reservations. Token estimates and delayed billing still limit how precisely that reservation predicts actual spend. Observability doesn't require logging every conversation. Record operational metadata by default, keep sensitive content capture deliberate and access-controlled, and use bounded labels for metrics. Individual request identifiers belong in traces or event records rather than metric dimensions.
Keep business decisions with the feature
The gateway can establish that a response completed and satisfies the operation’s output contract. The support feature still owns what happens next. A summary might be suitable for an agent to review but unsuitable for an automatic customer email. That decision depends on the workflow and its consequences. Similarly, a gateway can report a refusal, timeout or validation failure without deciding whether the application should queue work, request human review or abandon the task. Keep those outcomes distinct so the feature can respond appropriately. This prevents the gateway from gradually becoming the owner of every workflow that happens to use AI.
Start with one operation you can explain
You don't need multiple providers, semantic routing and a separate deployment to justify a gateway boundary. Start with one supported operation, one approved route, an identifiable prompt version, a deadline and useful execution records. Make it possible to explain a request from entry to final outcome. Before adding a second route, establish how you will evaluate it and which failures permit its use. Before adding retries, establish the attempt budget. Before promising a spending limit, establish how admission works under concurrency. The useful test is whether you can explain why a request used a particular model, what configuration shaped its answer, how much work was attempted and why the final outcome was accepted. A provider interface helps you make the call. The surrounding design lets you operate it.
Join the conversation! Your thoughts help the community grow.