Building a Simple Multi-Model LLM Gateway in ASP.NET Core

Introduction

Many AI applications begin with a single model provider. The first integration is usually straightforward: configure an API key, send a prompt, and return the response.

The architecture becomes more difficult when an application needs multiple providers. Different models may be suitable for different tasks, while providers may have different latency, pricing, rate limits, and availability.

A small gateway layer can centralize these decisions. The application sends one request to the gateway, and the gateway selects a provider, applies policy, and records the result.

This article demonstrates a simplified routing pattern in ASP.NET Core. The example is intentionally provider-neutral and is designed to explain the architecture rather than provide a production SDK.

Request model

The request contains the prompt and the type of work being performed.

public sealed record LlmRequest(
    string Prompt,
    string TaskType);

TaskType could be values such as interactive, reasoning, or classification. In a real application, this value might come from a workflow or service policy.

Provider abstraction

The gateway should not depend directly on one vendor SDK. An interface keeps provider-specific code behind an adapter.

public interface ILlmProvider
{
    string Name { get; }

    Task<string> CompleteAsync(
        string prompt,
        CancellationToken cancellationToken);
}

Each provider adapter implements this interface. This makes it possible to replace or reorder providers without changing application code.

Routing service

The router selects a provider based on the task type. The example below also demonstrates a simple fallback sequence.

public sealed class LlmRouter
{
    private readonly IReadOnlyDictionary<string, ILlmProvider> _providers;

    public LlmRouter(IEnumerable<ILlmProvider> providers)
    {
        _providers = providers.ToDictionary(
            provider => provider.Name,
            StringComparer.OrdinalIgnoreCase);
    }

    public async Task<string> RouteAsync(
        LlmRequest request,
        CancellationToken cancellationToken)
    {
        var route = request.TaskType.ToLowerInvariant() switch
        {
            "reasoning" => new[] { "primary-reasoning", "backup-general" },
            "classification" => new[] { "low-cost", "backup-general" },
            _ => new[] { "fast", "backup-general" }
        };

        foreach (var providerName in route)
        {
            if (!_providers.TryGetValue(providerName, out var provider))
            {
                continue;
            }

            try
            {
                return await provider.CompleteAsync(
                    request.Prompt,
                    cancellationToken);
            }
            catch (HttpRequestException)
            {
                // Continue to the next provider when the current one fails.
            }
        }

        throw new InvalidOperationException(
            "No configured LLM provider could process the request.");
    }
}

The fallback sequence should be expanded in production. Teams may also need timeouts, retry limits, circuit breakers, and provider-specific error handling.

Adding policy checks

Routing alone is not enough for a production system. Before sending a request, the gateway can validate:

Whether the application is authorized
Whether the request is within its budget
Whether sensitive data should be redacted
Which models are permitted for the workload
Which events should be logged

These checks are easier to maintain when they are implemented once in the gateway instead of separately in every application service.

Observability

A useful gateway should record structured telemetry for each request. Typical fields include:

Provider and model selected
Request timestamp
Response latency
Token usage
Retry or fallback count
Error category
Application or team identifier
Estimated cost

Avoid logging raw prompts by default. Redaction, retention, and access policies should be defined before enabling detailed request logging.

Where a managed gateway fits

Teams that do not want to maintain provider adapters and routing infrastructure themselves may evaluate a managed solution. nRouter’s company and architecture overview describes a managed LLM gateway with one API key, access to 151+ models, smart routing, budget enforcement, guardrails, PII redaction, provider failover, and spend attribution.

The appropriate choice depends on the organization’s security requirements, deployment model, provider contracts, and operational responsibilities.

Conclusion

A multi-model application becomes easier to evolve when model operations are separated from product logic. A gateway can provide one integration point for routing, fallback, policy enforcement, and observability.

The sample implementation is intentionally small, but the design principles apply to larger systems:

Hide provider-specific code behind adapters.
Make routing rules explicit.
Apply budget and security checks before provider calls.
Define fallback behavior and failure limits.
Capture enough telemetry to understand reliability and cost.

The goal is not to remove model differences. It is to manage those differences in one understandable and testable layer.