Microsoft Foundry Model Router: Keeping the Same Model Across a Conversation with Session Affinity

Choosing one AI model for an application is becoming harder.

A simple question may not need the same model as a complex reasoning task. If an application sends thousands of requests, using a powerful model for every request can also increase cost unnecessarily.

Microsoft Foundry Model Router is designed to solve that problem. Instead of hard-coding one underlying model, you deploy Model Router and let it choose an eligible model for each request based on the configured routing mode and model pool.

That works well for many workloads.

There is one problem, though.

A conversation contains multiple related requests. If the router evaluates every turn independently, the first message might go to one model while the next message goes to another.

For some applications, that is perfectly acceptable. For others, keeping the same underlying model across related turns can be useful.

Microsoft Foundry now provides session affinity for Chat Completions in preview. It allows an application to provide an opaque session ID and ask Model Router to try the same eligible model first for subsequent requests in that session.

This does not create a permanent lock to one model. Fallback, eligibility, availability, quota, policy, and safety requirements can still cause the router to switch models.

That distinction is important.

What Is Microsoft Foundry Model Router?

Model Router is deployed as a model in Microsoft Foundry.

Your application sends a request to the Model Router deployment instead of directly choosing an underlying model.

The router evaluates the request and selects an eligible model from its routing pool.

Conceptually, the flow looks like this:

Application
     |
     v
Model Router
     |
     +----> Model A
     |
     +----> Model B
     |
     +----> Model C
     |
     v
Response

The application still talks to one deployment.

The underlying model can change from request to request.

Microsoft currently provides routing modes such as:

• Balanced

• Quality

• Cost

Balanced is the default mode.

Quality is intended for workloads where response quality is more important than minimizing model cost.

Cost is useful for high-volume workloads where less expensive models can handle a significant portion of the traffic.

The router can also use a custom subset of models instead of the full supported pool.

Why Can the Model Change Between Conversation Turns?

Chat Completions is stateless.

The application normally sends the conversation history with every request.

For example:

Turn 1
User: Help me plan a trip to London.

Turn 2
User: Make it suitable for a family.

Turn 3
User: Add restaurant suggestions.

The application sends the relevant conversation history each time.

Without session affinity, Model Router can evaluate each request independently.

The result could look like:

Turn 1 -> Model A
Turn 2 -> Model B
Turn 3 -> Model A

This is not necessarily a problem.

The models are selected because the router considers them appropriate for the individual requests.

But some applications benefit from more consistency.

For example:

• Long-running assistants

• Coding assistants

• Customer support conversations

• Planning applications

• Multi-turn research workflows

• Applications where prompt caching matters

In these cases, repeatedly using the same eligible model can be useful.

What Is Session Affinity?

Session affinity gives Model Router an application-owned session identifier.

The application tells the router:

These requests belong to the same conversation.

The router can then associate the session with the model that successfully served the request.

For later requests using the same session ID, the router attempts to use that associated eligible model first.

The important word here is attempts.

Session affinity is not a guarantee that every request will use exactly the same model.

The basic behavior is:

First request
     |
     v
Model Router selects Model A
     |
     v
Session associated with Model A

Second request
     |
     v
Same session ID
     |
     v
Try Model A first
     |
     +---- eligible and available ---> Model A
     |
     +---- unavailable/ineligible ---> Another eligible model

Microsoft describes this as a sticky routing mode.

Session Affinity Is Best-Effort

This is probably the most important thing to understand.

Session affinity does not mean:

Session ID = permanent model lock

Instead, it means:

Session ID = try the associated eligible model first

Suppose the first request uses:

Model A

The next request normally attempts Model A first.

But if Model A cannot serve the request because of eligibility, availability, quota, fallback, policy, or another requirement, the router can use another model.

For example:

First turn
Model A -> 200 OK

Second turn
Model A -> 429
Model B -> 200 OK

The second response comes from Model B.

This is reported as a session-affinity decision of:

switch

The fallback behavior remains important because session affinity should not prevent the service from handling a request when the associated model cannot serve it.

How to Configure Session Affinity

The current preview uses the ModelRouterControls=V1Preview feature header with Chat Completions.

In Python:

import uuid

from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={
        "Foundry-Features": "ModelRouterControls=V1Preview"
    },
)

session_id = f"trip-planner-{uuid.uuid4()}"

session_affinity = {
    "routing_config": {
        "session_affinity": {
            "mode": "sticky",
            "session_id": session_id,
        }
    }
}

The session ID should be an opaque application-owned value.

Do not put:

Passwords
API keys
Access tokens
Personal information

inside the session ID.

A random identifier is a better choice.

For example:

session_id = f"conversation-{uuid.uuid4()}"

The same ID should then be used for each turn belonging to that conversation.

Sending the First Conversation Turn

Suppose we start with:

messages = [
    {
        "role": "system",
        "content": "You are a helpful travel assistant."
    },
    {
        "role": "user",
        "content": "Plan a three-day trip to London."
    }
]

Send the request through the Model Router deployment:

first_response = client.chat.completions.create(
    model=deployment,
    messages=messages,
    extra_body=session_affinity,
)

The router selects an eligible model.

For example:

Selected model: Model A
Affinity decision: initialize

The first successful request establishes the model association for the session.

Sending the Second Turn

Now the user asks:

Add restaurant suggestions and indoor activities.

The application needs to keep the conversation history and use the same session ID.

messages.extend(
    [
        {
            "role": "assistant",
            "content": first_response.choices[0].message.content
        },
        {
            "role": "user",
            "content": "Add restaurant suggestions and indoor activities."
        }
    ]
)

Then send:

second_response = client.chat.completions.create(
    model=deployment,
    messages=messages,
    extra_body=session_affinity,
)

Because the same session ID is being used, Model Router can attempt to retain the previously associated model.

A successful result could look like:

First turn
Model: Model A
Affinity: initialize

Second turn
Model: Model A
Affinity: retain

This is the behavior you generally want when maintaining model consistency across a conversation.

Understanding initialize, retain, and switch

The response can include session-affinity metadata.

The important decisions are:

For example:

{
  "session_affinity": {
    "mode": "sticky",
    "source": "session_id_payload",
    "decision": "retain"
  }
}

This tells you that the request used sticky session affinity and the associated model was retained.

A switch decision should not automatically be treated as an error.

It can mean the original model could not handle that particular request and Model Router successfully selected another eligible model.

Checking Which Model Actually Served the Request

Even when using session affinity, it is useful to inspect the response.

The Chat Completions response contains a model field identifying the underlying model that served the request.

For example:

print("Serving model:", response.model)

You can also inspect the routing details when the preview metadata is available:

model_selection_details = getattr(
    response,
    "model_selection_details",
    None
)

if model_selection_details:
    router_details = model_selection_details.get(
        "model_router_details",
        {}
    )

    affinity_details = router_details.get(
        "session_affinity"
    )

    if affinity_details:
        print(
            "Affinity mode:",
            affinity_details.get("mode")
        )

        print(
            "Affinity source:",
            affinity_details.get("source")
        )

        print(
            "Affinity decision:",
            affinity_details.get("decision")
        )

This is useful for troubleshooting because you can distinguish:

Model changed because routing selected another model

from:

Model changed because session affinity had to switch

Session IDs Should Be Managed by Your Application

Model Router does not need to know what your application calls a conversation.

Your application can maintain something like:

Conversation
ID: 8f4d...

User
    |
    +-- Turn 1
    +-- Turn 2
    +-- Turn 3
    +-- Turn 4

The application maps that conversation to:

session_id = 8f4d...

Every related Chat Completions request uses that identifier.

For another conversation:

session_id = 93ac...

Use a different value.

Do not reuse one session ID for unrelated users or unrelated conversations.

Session Affinity Does Not Store Your Conversation

The session ID is an association mechanism.

It does not mean Model Router becomes your conversation database.

Your application still owns the conversation history.

For example:

messages = [
    {"role": "system", "content": "..."},
    {"role": "user", "content": "..."},
    {"role": "assistant", "content": "..."},
    {"role": "user", "content": "..."}
]

Your application continues to send the required history with each Chat Completions request.

Session affinity only helps Model Router associate related requests with an underlying model.

This distinction is important when designing the application architecture.

Session Affinity and Prompt Caching

There is another reason session affinity can be useful.

Consecutive conversation requests often contain overlapping prompt prefixes.

For example:

System instructions
+
Conversation history
+
New user message

If the same model handles consecutive requests, there can be a better opportunity for prompt-cache reuse.

However, session affinity does not guarantee a cache hit.

The router does not use session affinity as a cache-aware model selection mechanism.

Think of it this way:

Session affinity
       ↓
Try the same eligible model
       ↓
Potentially better cache reuse

not:

Session affinity
       ↓
Guaranteed cached prompt

The actual cache behavior depends on the service and request conditions.

The Association Expires

A session association is not permanent.

Microsoft's current documentation states that the model association expires after 30 minutes without a successful create or update.

That means applications should not assume that a session ID permanently maps to one model.

For example:

10:00  Model A
10:10  Model A
10:20  Model A

The association can remain active while successful requests continue updating it.

But after a sufficiently long period without a successful request:

10:20  Last successful request

11:00  New request

the application may need to establish a new association.

This is another reason not to treat session affinity as a permanent model lock.

What Happens During a Model Failure?

Imagine a conversation is associated with:

Model A

A later request arrives.

Model Router tries Model A, but the request gets a retryable failure:

Model A -> 429

The router can then try another eligible model:

Model B -> 200

The response can report:

Affinity decision: switch

The routing trace can show both attempts.

Conceptually:

Session
  |
  v
Associated Model A
  |
  | 429
  v
Fallback Model B
  |
  | 200
  v
Response

This is a useful design because session consistency should not come at the expense of availability.

When Should You Use Session Affinity?

Session affinity makes the most sense when conversation-level model consistency provides a real benefit.

Long-Running Assistants

A user may interact with an assistant over many turns.

Keeping the same model when possible can make the behavior more predictable.

Coding Assistants

A coding conversation may contain:

Architecture discussion
      ↓
Code generation
      ↓
Bug analysis
      ↓
Refactoring
      ↓
Testing

Keeping the same eligible model can be useful for maintaining consistent behavior across the conversation.

Customer Support

Support applications often have a long conversation history.

Switching models on every turn is not necessarily bad, but some teams may prefer a more consistent serving model.

Planning Applications

Travel, project planning, research, and similar workflows can have many related turns.

Session affinity can help keep the serving model stable while still allowing fallback when necessary.

When You May Not Need It

Do not enable session affinity just because it exists.

For independent requests such as:

Classify this email.
Summarize this document.
Translate this sentence.
Extract these fields.

there may be little benefit.

If each request stands alone, independent routing can be exactly what you want.

The router can choose the most suitable model for each request.

In that situation:

Request 1 -> Model A
Request 2 -> Model C
Request 3 -> Model B

may be perfectly reasonable.

Session Affinity vs Fixed Model Deployment

There are two different strategies.

Fixed Model

You explicitly choose:

Model A

Every request goes to Model A unless your application changes the deployment.

This gives strong predictability.

Model Router With Session Affinity

You choose:

Model Router

and provide a session ID.

The router tries to keep the conversation on the same eligible model, but it can switch when necessary.

The comparison looks like this:

If you absolutely require one specific model for every request, use a direct model deployment instead.

If you want automated routing while improving consistency across a conversation, session affinity is a better fit.

You Can Disable Affinity for a Request

There are cases where you may want normal routing for one particular request.

The session-affinity mode can be set to:

none

For example:

normal_routing = {
    "routing_config": {
        "session_affinity": {
            "mode": "none"
        }
    }
}

That request uses normal routing and does not read or update the model association.

This can be useful when a specific operation is intentionally independent of the existing conversation routing behavior.

Session Affinity Through a Request Header

Instead of putting the session ID in the request body, the application can provide it through:

x-ms-session-id

The application-owned identifier can therefore be supplied in either location.

If both a valid body identifier and a valid header identifier are present, the body value takes precedence.

This can be useful if your application infrastructure already manages session identifiers at the HTTP layer.

For example:

HTTP Request
    |
    +-- x-ms-session-id: conversation-123
    |
    +-- Chat Completions body

Keep the identifier opaque and avoid putting sensitive information into it.

What If the Session ID Is Invalid?

Session IDs have validation requirements.

The identifier must contain between 1 and 256 Unicode code points and include at least one non-whitespace character.

If the body contains an invalid identifier but the header contains a valid identifier, Model Router can use the valid header value.

If neither is valid, Chat Completions falls back to normal routing.

This is another reason to generate session IDs rather than constructing them from arbitrary user input.

Monitoring Session Affinity

When working with a routing system, logging only the final response is not enough.

You should ideally capture:

Conversation ID
Session ID reference
Serving model
Routing mode
Affinity decision
Request latency
Fallback information
HTTP status

Do not log the actual sensitive contents of the session ID if your design includes sensitive values. Better yet, make the session ID opaque from the beginning.

For preview routing metadata, Model Router can expose details such as:

model
routing mode
routing trace
attempted models
HTTP status
latency
session affinity decision

The exact metadata is preview functionality, so applications should handle missing fields gracefully.

A Better Diagnostic Pattern

A production application should not assume that every response contains complete routing metadata.

Use defensive code:

model_selection_details = getattr(
    response,
    "model_selection_details",
    None
)

if not model_selection_details:
    print("Routing metadata unavailable")
else:
    router_details = model_selection_details.get(
        "model_router_details",
        {}
    )

    affinity = router_details.get("session_affinity")

    if affinity:
        print("Affinity:", affinity)

This matters because session-affinity lookup or persistence can fail without causing inference itself to fail.

In that situation, Model Router can continue with normal routing and the complete affinity object may not be present.

Do Not Treat switch as a Failure

Suppose you receive:

Affinity decision: switch

That does not necessarily mean the request failed.

It can mean:

Associated model
      ↓
Not eligible or unavailable
      ↓
Fallback model
      ↓
Successful response

The request can still be successful.

The important question is:

Why did the router switch?

For that, inspect the routing trace and the top-level model field when the preview metadata is available.

Model Router Still Needs Evaluation

Using Model Router does not remove the need for application-level evaluation.

Before moving a production workload to Model Router, compare it with your current baseline.

At minimum, measure:

Quality
Cost
Latency

Use prompts that represent the real workload.

For example:

100+ representative prompts

can give you a much better signal than a handful of manually selected examples.

Also separate different workload categories.

For example:

Coding
Summarization
Classification
Reasoning
Customer support
Extraction

A router that performs well for one category may behave differently for another.

Microsoft's current evaluation guidance recommends comparing representative workloads rather than relying on a generic quality threshold.

Common Mistakes

Treating Session Affinity as a Hard Lock

It is not.

The router can switch models when required.

Generating a New Session ID for Every Request

If every turn gets a different session ID, the router cannot associate those turns as one session.

Use one session ID for one conversation.

Reusing One Session ID for Everyone

That mixes unrelated conversations.

Generate separate identifiers.

Putting Personal Information in the Session ID

Avoid values such as:

[email protected]

or:

customer-987654

Use opaque identifiers instead.

Forgetting Conversation History

Session affinity does not turn Chat Completions into a stateful conversation store.

Your application still needs to send the appropriate conversation history.

Assuming a Cache Hit

Session affinity may improve the opportunity for cache reuse, but it does not guarantee caching.

Ignoring switch

A model switch can be normal fallback behavior.

Investigate the reason before treating it as a bug.

Depending on Preview Metadata Without Fallback Handling

Preview response fields can be absent.

Write code that works even when routing metadata is unavailable.

A Practical Architecture

A typical application can use this structure:

                    User
                      |
                      v
              Application API
                      |
                      +---- Conversation ID
                      |
                      +---- Session ID
                      |
                      v
             Foundry Model Router
                      |
              +-------+-------+
              |       |       |
              v       v       v
            Model A Model B Model C
              |
              v
           Response
              |
              v
        Application stores
        conversation history

The application owns:

Conversation state
Session ID
Authentication
Authorization
Business logic

Model Router owns:

Per-request model selection
Routing policy
Eligible model selection
Fallback behavior

That separation keeps the architecture clean.

A Complete Minimal Example

Here is a simplified Python example:

import os
import uuid

from openai import AzureOpenAI

endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
api_key = os.environ["AZURE_OPENAI_API_KEY"]
deployment = os.environ["MODEL_ROUTER_DEPLOYMENT"]

client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={
        "Foundry-Features": "ModelRouterControls=V1Preview"
    },
)

session_id = f"conversation-{uuid.uuid4()}"

session_affinity = {
    "routing_config": {
        "session_affinity": {
            "mode": "sticky",
            "session_id": session_id,
        }
    }
}

messages = [
    {
        "role": "system",
        "content": "You are a helpful travel assistant."
    },
    {
        "role": "user",
        "content": "Plan a three-day trip to London."
    }
]

first_response = client.chat.completions.create(
    model=deployment,
    messages=messages,
    extra_body=session_affinity,
)

messages.extend(
    [
        {
            "role": "assistant",
            "content": first_response.choices[0].message.content,
        },
        {
            "role": "user",
            "content": "Add restaurant suggestions and indoor activities.",
        },
    ]
)

second_response = client.chat.completions.create(
    model=deployment,
    messages=messages,
    extra_body=session_affinity,
)

print("First model:", first_response.model)
print("Second model:", second_response.model)

In a real application, you would also add error handling, secure credential management, logging, and appropriate conversation storage.

Summary

Microsoft Foundry Model Router can automatically choose an underlying AI model for each request, helping applications balance quality, cost, and performance.

For multi-turn Chat Completions applications, the new session-affinity preview adds another option. By sending the same application-owned session ID with related requests, the router can try to keep those requests on the same eligible model.

The first successful request normally establishes the association. Later requests can report retain when the same model is used or switch when another model is required.

The key point is that session affinity is best-effort, not a permanent model lock. If the associated model is unavailable or no longer eligible, Model Router can fall back to another model.

For developers, the safest design is simple: generate an opaque session ID for each conversation, reuse it for related turns, keep the conversation history in your own application, monitor the actual serving model, and handle model switches normally.