Microsoft Foundry Model Router: Tracking Fallbacks and Routing Latency Per Request

Modern AI applications rarely depend on a single model. A production workload may use different models for simple questions, complex reasoning, coding, tool calling, or latency-sensitive requests. Microsoft Foundry Model Router is designed to handle this model selection automatically.

That flexibility becomes especially useful when a selected model is temporarily unavailable or experiences an endpoint problem. Model Router can automatically fail over to another eligible model instead of forcing the application to implement its own retry and routing logic.

But automatic routing introduces an operational question: How do you know what happened to an individual request?

For production systems, simply knowing that a request succeeded is not enough. Developers may need to know which model served the request, whether another model was attempted first, how long the routing decision took, and whether a fallback occurred.

Microsoft Foundry now provides per-request routing metadata in preview for Chat Completions. This makes it possible to inspect routing behavior at the individual request level and use that information for troubleshooting, observability, and performance analysis.

What Is Microsoft Foundry Model Router?

Model Router is an optimization layer that evaluates an incoming request and selects an eligible model based on the configured routing mode and model pool.

Instead of changing the model name in application code, an application sends requests to the Model Router deployment. The router determines which underlying model should handle each request.

A simplified architecture looks like this:

Application
    |
    v
Model Router
    |
    +---- Model A
    |
    +---- Model B
    |
    +---- Model C

The important difference from traditional load balancing is that routing is based on the characteristics of the request rather than simply distributing requests evenly.

For example:

A simple classification request may be sent to a faster model.
A complex reasoning request may be sent to a more capable model.
A tool-heavy request may be routed to a model that is better suited for that workload.
If the selected model encounters a transient endpoint problem, Model Router can attempt another eligible model.

This means model selection can vary from request to request, even when all requests come through the same application endpoint.

Why Request-Level Routing Observability Matters

Automatic routing is useful, but it can make production behavior harder to understand.

Imagine an application where a user reports that one request took noticeably longer than another. If both requests were sent to the same Model Router deployment, the application logs might initially show only a successful response.

Without routing information, several questions remain unanswered:

Which underlying model handled the request?
Did Model Router initially select another model?
Did a fallback happen?
How many models were attempted?
How long did the routing decision take?
Which attempt ultimately succeeded?
Did the failed attempt return an HTTP error?

These details can be important when investigating latency spikes or intermittent failures.

Per-request routing metadata provides additional visibility into these decisions.

Enabling Per-Request Routing Metadata

The current preview implementation exposes routing information through the Chat Completions response when the preview feature is enabled.

For Python applications, the Azure OpenAI client can request the routing metadata by supplying the appropriate feature header.

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={
        "Foundry-Features": "ModelRouterControls=V1Preview"
    }
)

The application can then send a normal Chat Completions request through the Model Router deployment.

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Explain dependency injection in .NET."
        }
    ]
)

The response can contain additional model-selection information when the preview contract returns it.

Because this is a preview capability, production applications should avoid assuming that every response will contain identical metadata.

Understanding the Routing Trace

The most useful part of the metadata is the routing trace.

A routing trace can show the routing decision and the ordered attempts made for that request.

A simplified example looks like this:

{
  "model_selection_details": {
    "model_router_details": {
      "mode": "balanced",
      "routing_trace": [
        {
          "latency_ms": 19,
          "attempts": [
            {
              "model": "model-a",
              "result": {
                "status": 404
              }
            },
            {
              "model": "model-b",
              "result": {
                "status": 200
              }
            }
          ]
        }
      ]
    }
  }
}

This tells us considerably more than a normal successful response.

In this example:

Model Router used Balanced mode.
The routing decision reported 19 milliseconds of routing latency.
The first model was attempted.
The first attempt returned an HTTP 404.
A second model was attempted.
The second model returned HTTP 200.
The second model ultimately served the request.

This is the type of information that helps distinguish a normal routed request from a request that required fallback.

Detecting an Automatic Fallback

A fallback occurs when the initially selected model cannot successfully handle the request and Model Router moves to another eligible model.

Conceptually, the request flow looks like this:

Request
   |
   v
Model Router
   |
   v
Model A
   |
   | Failure
   v
Model B
   |
   | Success
   v
Response

From an application's perspective, the request may still succeed without custom retry code.

However, operationally, the request was different from a normal single-attempt request.

That distinction matters.

A small number of fallbacks may be expected during transient service conditions. A growing fallback rate, however, can indicate an underlying availability or configuration problem.

For example:

Requests:              100,000
Successful responses:   99,950
Requests with fallback:    1,200

The overall success rate may look healthy, but the fallback count tells a different operational story.

Tracking the Serving Model

The response model field identifies the model that ultimately handled the request.

For example:

serving_model = response.model

print(f"Serving model: {serving_model}")

This can be included in application telemetry.

A useful log record might contain:

Request ID
Timestamp
Serving model
Routing mode
Routing latency
Attempt count
Fallback detected
Response latency
Token usage
HTTP status

This creates a much more useful operational record than simply logging whether the API request succeeded.

Measuring Routing Latency

Routing latency is different from the total response latency.

Consider this simplified request:

Client
  |
  |---- Router decision: 20 ms
  |
  |---- Model inference: 850 ms
  |
  v
Response

The routing decision contributes only a small portion of the overall request duration.

This distinction becomes important when analyzing performance.

Suppose an application's p95 latency increases from 1.2 seconds to 2.1 seconds.

The routing trace can help determine whether the additional latency is associated with routing or with model execution and fallback behavior.

For production monitoring, it is useful to track at least:

Metric	Why It Matters
Total request latency	Measures end-to-end user experience
Routing latency	Measures overhead introduced by routing
Model response latency	Shows model execution performance
Fallback rate	Indicates how often requests require another model
Attempt count	Shows how many models were tried
Serving model distribution	Shows which models are handling traffic
Error rate	Identifies failed requests

Do not treat average latency as the only performance metric. Median and tail latency such as p90 or p95 are often more useful for understanding real production behavior.

Building a Simple Routing Log

The application can extract the serving model and record it with other request information.

For example:

import time

start_time = time.perf_counter()

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {
            "role": "user",
            "content": "Explain async programming in C#."
        }
    ]
)

elapsed_ms = (time.perf_counter() - start_time) * 1000

print({
    "serving_model": response.model,
    "response_latency_ms": round(elapsed_ms, 2)
})

The response metadata can then be inspected when available to determine whether routing involved multiple attempts.

For production systems, this information should normally be sent to the application's existing observability platform rather than printed to the console.

Logging Fallbacks Without Creating Noise

It is tempting to log every field returned by the router. That can quickly make application telemetry difficult to use.

A better approach is to normalize the information into a small operational record.

For example:

routing_event = {
    "serving_model": response.model,
    "fallback": False,
    "attempt_count": 1,
    "routing_latency_ms": None
}

When routing metadata indicates multiple attempts:

routing_event = {
    "serving_model": response.model,
    "fallback": True,
    "attempt_count": 2,
    "routing_latency_ms": 19
}

The exact parsing logic should account for missing metadata because preview response fields may not be present for every request.

This also prevents the application from treating absent routing metadata as evidence that no routing occurred.

Model Subsets and Fallback Behavior

Model Router can also be configured with a specific subset of models.

This is important for organizations that have restrictions around:

Approved models
Data residency
Compliance
Cost
Performance
Application compatibility

The configured model subset also limits the models that can participate in fallback.

For example:

Allowed Model Subset

Model A
Model B
Model C

If Model A becomes unavailable, the router can select another eligible model from that configured set.

This is safer than allowing fallback to an arbitrary model that the application has not approved.

For workloads that depend on fallback, the subset should contain at least two eligible models.

A Practical Observability Dashboard

For a production AI application, routing information becomes more valuable when aggregated.

A dashboard could contain:

Model Router Production Overview

Total Requests              1,250,000
Fallback Requests               8,420
Fallback Rate                    0.67%
Average Routing Latency          18 ms
P95 Request Latency            1.42 s
Error Rate                       0.12%

Serving Model Distribution
--------------------------------
Model A                          52%
Model B                          31%
Model C                          17%

Another useful visualization is fallback rate over time.

Fallback Rate

2.0% |                 *
1.5% |                **
1.0% |       **      ***
0.5% |******* ** ******
0.0% |____________________
       09:00  10:00  11:00

A sudden increase can trigger investigation even when the application's overall success rate remains high.

Troubleshooting a Latency Spike

Suppose users report that an AI feature suddenly became slower.

Start with total latency:

P50:  620 ms
P95:  2.8 s
P99:  5.4 s

Next, inspect routing behavior.

If requests with high latency also show multiple model attempts, fallback may be contributing to the tail latency.

For example:

Normal request
Router: 18 ms
Model A: 650 ms
Total: 668 ms

Compared with:

Fallback request
Router: 21 ms
Model A: failed
Model B: 1,400 ms
Total: ~1,421 ms

The routing metadata provides evidence that the slower request followed a different execution path.

This is much more actionable than simply observing that the API response was slow.

Common Mistakes

Assuming Every Successful Request Used One Model

A successful response does not necessarily mean the first selected model completed the request.

Inspect the routing information when troubleshooting unexpected behavior.

Treating Missing Metadata as No Fallback

Preview metadata may not be returned for every request.

Do not automatically classify missing routing information as a successful single-model attempt.

Monitoring Only Average Latency

An average can hide tail behavior.

Track p50, p90, p95, or p99 latency for user-facing applications where occasional slow requests matter.

Ignoring Model Distribution

If one model unexpectedly handles a much larger percentage of requests, the change can affect both cost and latency.

Track the serving model distribution over time.

Allowing Unapproved Models in a Fallback Pool

Fallback should remain within the models that the workload is permitted to use.

Use model subsets when compliance, cost, or compatibility requirements make unrestricted routing inappropriate.

Treating Router Latency as Total Model Latency

Routing latency measures the routing decision. It should not be confused with the total time spent generating the response.

Best Practices for Production

A practical Model Router observability strategy should include the following:

Log the serving model for every request where the information is available.
Track fallback frequency rather than looking only at successful responses.
Measure routing and total latency separately.
Monitor tail latency, especially p95 and p99.
Track model distribution to understand how traffic is being routed.
Use model subsets when application or compliance requirements restrict eligible models.
Keep fallback telemetry separate from application errors. A fallback can still result in a successful request.
Treat preview metadata defensively because response fields can change.
Correlate routing events with application request IDs so slow or failed requests can be investigated end to end.
Reevaluate routing behavior after model or configuration changes.

When Direct Model Deployments Still Make Sense

Model Router is useful when an application benefits from dynamic model selection, but it is not the right choice for every request.

A direct model deployment can be preferable when an application requires:

Deterministic model selection
Model-specific parameters
Strict compatibility requirements
A compliance-mandated model
Reproducible performance characteristics

A practical architecture can use both approaches.

Application
    |
    +---- General workloads ----> Model Router
    |
    +---- Specialized workload -> Direct Model
    |
    +---- Compliance workload --> Approved Model

This hybrid approach allows general traffic to benefit from dynamic routing while keeping strict workloads under explicit model control.

Conclusion

Microsoft Foundry Model Router removes much of the application-level complexity involved in selecting and failing over between AI models. However, automatic routing should not become a black box for production teams.

Per-request routing metadata provides a way to understand what happened to individual Chat Completions requests, including the serving model, routing behavior, latency information, and ordered model attempts when fallback occurs.

The most important operational lesson is simple: a successful request does not always mean a simple request.

A request may have been routed to one model, failed, and then completed successfully through another. Without request-level observability, that behavior can remain invisible.

By tracking serving-model distribution, routing latency, fallback frequency, attempt counts, and tail latency, teams can turn Model Router from an opaque optimization layer into an observable part of their AI infrastructure.

That visibility is especially valuable when diagnosing production latency, unexpected model usage, intermittent failures, and changes in application cost or performance.