Building a Resilient Multi-Model LLM Gateway in .NET

Introduction

Connecting a .NET application to a single large language model (LLM) provider is relatively straightforward. The architecture becomes more complicated when an application needs multiple providers, fallback models, rate limits, budget controls, guardrails, and centralized monitoring.

One approach is to implement all of this logic directly inside the application. However, that quickly leads to provider-specific code being spread across controllers, services, background jobs, and agents.

A better approach is to place an LLM gateway between the application and model providers.

The application communicates with one consistent interface, while the gateway handles model routing, reliability, security, cost controls, and observability.

What Is an LLM Gateway?

An LLM gateway is an intermediary layer between applications and one or more AI model providers.

Instead of an application directly calling every provider, the architecture looks like this:

.NET Application
       |
       v
   LLM Gateway
       |
   +---+---+----------------+
   |       |                |
   v       v                v
Provider A Provider B   Provider C
   |       |                |
 Model 1  Model 2         Model 3

The gateway provides a consistent entry point while hiding provider-specific implementation details from the application.

Why Use an LLM Gateway?

A gateway becomes useful when an application has more than a simple model call.

Typical responsibilities include:

Model and provider routing
Automatic fallback
Rate limiting
Budget enforcement
Prompt and response guardrails
Usage and cost tracking
Centralized logging and tracing
Provider credential management

This separation allows application developers to focus on business functionality instead of repeatedly implementing infrastructure logic for each model provider.

Basic Request Flow

A production LLM request can follow a sequence such as:

Application
    |
    v
Authenticate Request
    |
    v
Check Rate Limit and Budget
    |
    v
Input Guardrails
    |
    v
Select Model
    |
    v
Call Provider
    |
    +---- Failure? ----> Retry / Fallback
    |
    v
Response Guardrails
    |
    v
Record Usage and Cost
    |
    v
Return Response

A typical request therefore involves more than simply sending a prompt to a model.

Step 1: Authenticate

The gateway first determines whether the calling application or user is authorized to make the request.

Step 2: Check Limits

The gateway checks whether the request is within the configured rate and budget limits.

Step 3: Apply Input Guardrails

The incoming request can be inspected for issues such as sensitive information, excessive input size, or disallowed content.

Step 4: Select a Model

The gateway chooses a provider and model according to routing rules.

For example:

Simple classification
        ↓
Lower-cost model

Complex reasoning
        ↓
More capable model

Step 5: Call the Provider

The gateway sends the request to the selected provider.

Step 6: Retry or Fail Over

If the provider experiences a temporary failure, the gateway can retry or select another compatible model.

Step 7: Apply Response Guardrails

The response can be checked before it is returned to the application.

Step 8: Record Telemetry

The gateway records information such as latency, token usage, estimated cost, errors, and fallback attempts.

Calling an OpenAI-Compatible Gateway from .NET

One advantage of an OpenAI-compatible gateway is that a .NET application can communicate with it using familiar HTTP request patterns.

The following example uses HttpClient:

using System.Net.Http.Headers;
using System.Net.Http.Json;

var apiKey = Environment.GetEnvironmentVariable("NROUTER_API_KEY");

using var client = new HttpClient
{
    BaseAddress = new Uri("https://api.nrouter.ai/")
};

client.DefaultRequestHeaders.Authorization =
    new AuthenticationHeaderValue("Bearer", apiKey);

var request = new
{
    model = "deepseek-v4-pro",
    messages = new[]
    {
        new
        {
            role = "user",
            content = "Explain semantic caching in simple terms."
        }
    }
};

var response = await client.PostAsJsonAsync(
    "v1/chat/completions",
    request
);

response.EnsureSuccessStatusCode();

var result = await response.Content.ReadAsStringAsync();

Console.WriteLine(result);

The important architectural point is that the application does not need to implement provider-specific authentication and routing logic throughout its codebase.

The gateway becomes the integration boundary.

Protect API Credentials

Production API keys should not be hard-coded in source code.

For example, the sample reads the key from an environment variable:

var apiKey =
    Environment.GetEnvironmentVariable("NROUTER_API_KEY");

Depending on the deployment environment, credentials can instead be stored in a secret manager or another managed configuration system.

Making the Gateway Resilient

A multi-model gateway is useful only if failure handling is designed carefully.

Not every error should trigger a retry.

A practical policy can distinguish between temporary and permanent failures.

Retry Temporary Failures

Transient network problems and temporary service failures may be suitable for retry.

Exponential backoff can prevent the application from immediately sending repeated requests:

Attempt 1 → wait
Attempt 2 → wait longer
Attempt 3 → wait longer

The gateway should also enforce a maximum number of attempts.

Respect Rate Limits

If a provider indicates that the client has exceeded its request limit, the gateway should respect the provider's rate-limit information rather than continuously retrying.

Fail Over When Appropriate

A gateway can route an eligible request to a backup provider when the primary provider is unavailable.

For example:

Primary:
Provider A / Model X

        |
        | Temporary failure
        v

Fallback:
Provider B / Model Y

However, the fallback model cannot simply be any available model.

It should be compatible with the application's requirements.

Important compatibility considerations include:

Context-window requirements
Required tools
Structured-output requirements
Supported input modalities
Response format
Model capabilities

Do Not Retry Everything

Authentication failures, malformed requests, and other permanent client-side errors generally should not be blindly retried.

A retry policy should therefore classify failures rather than treating every non-success response identically.

Enforcing Budgets Before Requests

Usage reporting tells an organization what it has already spent.

Budget enforcement serves a different purpose: it prevents or limits additional spending once a configured threshold has been reached.

Budgets can be applied at different levels, such as:

Organization
Team
Application
Environment
API key
Individual agent

A simplified flow is:

Incoming Request
       |
       v
Check Current Usage
       |
       v
Budget Available?
    /       \
  Yes        No
   |          |
   v          v
Continue    Reject /
            Reroute

This becomes especially important for autonomous agents.

An agent can potentially generate multiple model requests without a person explicitly initiating every request. Without appropriate limits, unexpected usage can accumulate quickly.

Centralizing Guardrails

Changing the model provider should not automatically change the application's safety and security policies.

Guardrails can therefore be centralized in the gateway.

Common controls include:

Sensitive-data detection and redaction
Prompt-injection checks
Content-safety policies
Maximum prompt size
Maximum response size
Approved-model restrictions
Regional data-processing requirements

A typical request can pass through input and output checks:

User Request
     |
     v
Input Guardrails
     |
     v
Model Provider
     |
     v
Output Guardrails
     |
     v
Application

Centralizing these policies also makes them easier to review and update.

Observability Requirements

LLM applications need more than traditional application logs.

For each request, the gateway should ideally capture operational metadata such as:

Selected provider
Selected model
Routing decision
Routing reason
Request duration
Input token usage
Output token usage
Estimated cost
Cache status
Retry attempts
Fallback attempts
Final response status

Correlation IDs can connect gateway activity with the originating application's distributed trace.

For example:

Application Trace
       |
       +-- Gateway Request
               |
               +-- Provider A Attempt
               |
               +-- Provider B Fallback

This makes it easier to understand what happened when a request takes longer than expected or produces an error.

Protect Sensitive Data in Logs

Observability should not become a new source of data leakage.

Raw prompts and responses may contain confidential information, credentials, personal data, or proprietary business information.

Logging should therefore be designed according to the application's data-classification and retention requirements.

Making Routing Explainable

A routing system should not behave like a black box from an operational perspective.

For example, instead of simply recording:

Model: Model-B

the gateway can record useful routing metadata such as:

Task: Classification
Selected Model: Model-B
Reason: Cost-optimized route
Fallback Available: Model-C

The exact metadata will depend on the gateway implementation, but the principle is important.

When a production incident occurs, engineers need to understand why a particular provider or model was selected.

A Practical Routing Strategy

A simple routing policy might look like this:

Request
   |
   +-- Classification?
   |       |
   |       +--> Low-cost model
   |
   +-- Complex reasoning?
   |       |
   |       +--> Higher-capability model
   |
   +-- Provider unavailable?
           |
           +--> Compatible fallback model

A real implementation can make the decision using factors such as:

Task type
Model capability
Estimated cost
Current provider availability
Latency requirements
Context size
Tool requirements
Organizational policies

The important point is that routing rules should be explicit enough to test and troubleshoot.

Common Mistakes

Putting Provider Logic Everywhere

If every service directly understands multiple providers, changing providers becomes expensive.

Keep provider-specific concerns behind the gateway boundary where possible.

Using an Incompatible Fallback

A fallback model may have different capabilities.

For example, a model that does not support a required tool or structured response format may not be a valid fallback.

Retrying Permanent Errors

Repeatedly retrying invalid requests wastes resources and can increase latency and cost.

Retry policies should distinguish transient failures from permanent failures.

Logging Sensitive Prompts

Logging every prompt and response without considering data sensitivity can create security and compliance problems.

Log the metadata required for operations and apply appropriate redaction and retention policies.

Enforcing Budgets After Usage Occurs

A usage dashboard that reports spending after the fact is not the same as a budget control.

If spending limits matter, the gateway needs a decision point before sending the request.

LLM Gateway vs Direct Provider Integration

A direct integration can be perfectly reasonable for a small application.

The architecture changes when the application needs multiple operational controls.

Requirement	Direct Integration	LLM Gateway
Single provider	Simple	Simple
Multiple providers	Application-managed	Centralized
Model routing	Application-managed	Gateway-managed
Fallback	Custom implementation	Centralized
Budget enforcement	Custom implementation	Centralized
Guardrails	Distributed	Centralized
Usage tracking	Application-specific	Centralized
Provider credentials	Application-managed	Gateway-managed

A gateway introduces another infrastructure component, so it should be adopted when its operational benefits justify that additional complexity.

Where an OpenAI-Compatible Gateway Fits

An OpenAI-compatible gateway can provide a familiar API surface while placing routing and infrastructure concerns behind that interface.

For example:

.NET Services
     |
     | OpenAI-compatible API
     v
LLM Gateway
     |
     +------ Provider A
     |
     +------ Provider B
     |
     +------ Provider C

In this architecture, applications can use one integration point while the gateway handles provider selection and related operational policies.

A managed service such as nRouter.ai can be used as an example of this model, but the same architectural principles apply to other gateway implementations.

Recommended Architecture

For a production .NET application, a practical architecture can look like:

                  +----------------+
                  | .NET Services  |
                  +-------+--------+
                          |
                          v
                  +---------------+
                  |  LLM Gateway  |
                  +-------+-------+
                          |
             +------------+------------+
             |            |            |
             v            v            v
        Provider A   Provider B   Provider C
             |            |            |
           Model A      Model B      Model C

Gateway responsibilities:
- Authentication
- Rate limiting
- Budget checks
- Routing
- Retries
- Fallback
- Guardrails
- Observability

This separates application functionality from LLM infrastructure concerns.

When Should You Use an LLM Gateway?

A gateway becomes particularly valuable when an application has several of the following requirements:

Multiple LLM providers
Multiple models
Provider fallback
Cost controls
Centralized security policies
Autonomous agents
Organization-wide usage tracking
Centralized observability
Frequent model changes

For a small application using one provider with minimal operational requirements, a direct integration may be simpler.

Conclusion

Adding a second AI model can turn a straightforward API integration into an infrastructure problem. Provider routing, fallback behavior, rate limits, budgets, guardrails, credentials, and observability all need to be handled consistently.

An LLM gateway provides a dedicated boundary between .NET applications and model providers. Applications can communicate through a stable interface while the gateway manages the operational complexity behind it.

The most important design principles are to make routing explainable, validate budgets before requests, use compatible fallback models, distinguish retryable from permanent failures, centralize guardrails, and capture useful telemetry without unnecessarily storing sensitive prompts or responses.

For applications that depend on multiple models or autonomous AI workloads, this separation can make the overall architecture easier to operate, monitor, and evolve.