Choosing one AI model for an application is becoming harder.
A simple question may not need the same model as a complex reasoning task. If an application sends thousands of requests, using a powerful model for every request can also increase cost unnecessarily.
Microsoft Foundry Model Router is designed to solve that problem. Instead of hard-coding one underlying model, you deploy Model Router and let it choose an eligible model for each request based on the configured routing mode and model pool.
That works well for many workloads.
There is one problem, though.
A conversation contains multiple related requests. If the router evaluates every turn independently, the first message might go to one model while the next message goes to another.
For some applications, that is perfectly acceptable. For others, keeping the same underlying model across related turns can be useful.
Microsoft Foundry now provides session affinity for Chat Completions in preview. It allows an application to provide an opaque session ID and ask Model Router to try the same eligible model first for subsequent requests in that session.
This does not create a permanent lock to one model. Fallback, eligibility, availability, quota, policy, and safety requirements can still cause the router to switch models.
That distinction is important.
What Is Microsoft Foundry Model Router?
Model Router is deployed as a model in Microsoft Foundry.
Your application sends a request to the Model Router deployment instead of directly choosing an underlying model.
The router evaluates the request and selects an eligible model from its routing pool.
Conceptually, the flow looks like this:
Application
|
v
Model Router
|
+----> Model A
|
+----> Model B
|
+----> Model C
|
v
Response
The application still talks to one deployment.
The underlying model can change from request to request.
Microsoft currently provides routing modes such as:
Balanced is the default mode.
Quality is intended for workloads where response quality is more important than minimizing model cost.
Cost is useful for high-volume workloads where less expensive models can handle a significant portion of the traffic.
The router can also use a custom subset of models instead of the full supported pool.
Why Can the Model Change Between Conversation Turns?
Chat Completions is stateless.
The application normally sends the conversation history with every request.
For example:
Turn 1
User: Help me plan a trip to London.
Turn 2
User: Make it suitable for a family.
Turn 3
User: Add restaurant suggestions.
The application sends the relevant conversation history each time.
Without session affinity, Model Router can evaluate each request independently.
The result could look like:
Turn 1 -> Model A
Turn 2 -> Model B
Turn 3 -> Model A
This is not necessarily a problem.
The models are selected because the router considers them appropriate for the individual requests.
But some applications benefit from more consistency.
For example:
Long-running assistants
Coding assistants
Customer support conversations
Planning applications
Multi-turn research workflows
Applications where prompt caching matters
In these cases, repeatedly using the same eligible model can be useful.
What Is Session Affinity?
Session affinity gives Model Router an application-owned session identifier.
The application tells the router:
These requests belong to the same conversation.
The router can then associate the session with the model that successfully served the request.
For later requests using the same session ID, the router attempts to use that associated eligible model first.
The important word here is attempts.
Session affinity is not a guarantee that every request will use exactly the same model.
The basic behavior is:
First request
|
v
Model Router selects Model A
|
v
Session associated with Model A
Second request
|
v
Same session ID
|
v
Try Model A first
|
+---- eligible and available ---> Model A
|
+---- unavailable/ineligible ---> Another eligible model
Microsoft describes this as a sticky routing mode.
Session Affinity Is Best-Effort
This is probably the most important thing to understand.
Session affinity does not mean:
Session ID = permanent model lock
Instead, it means:
Session ID = try the associated eligible model first
Suppose the first request uses:
Model A
The next request normally attempts Model A first.
But if Model A cannot serve the request because of eligibility, availability, quota, fallback, policy, or another requirement, the router can use another model.
For example:
First turn
Model A -> 200 OK
Second turn
Model A -> 429
Model B -> 200 OK
The second response comes from Model B.
This is reported as a session-affinity decision of:
switch
The fallback behavior remains important because session affinity should not prevent the service from handling a request when the associated model cannot serve it.
How to Configure Session Affinity
The current preview uses the ModelRouterControls=V1Preview feature header with Chat Completions.
In Python:
import uuid
from openai import AzureOpenAI
client = AzureOpenAI(
azure_endpoint=endpoint,
api_key=api_key,
api_version="2024-10-21",
default_headers={
"Foundry-Features": "ModelRouterControls=V1Preview"
},
)
session_id = f"trip-planner-{uuid.uuid4()}"
session_affinity = {
"routing_config": {
"session_affinity": {
"mode": "sticky",
"session_id": session_id,
}
}
}
The session ID should be an opaque application-owned value.
Do not put:
Passwords
API keys
Access tokens
Personal information
inside the session ID.
A random identifier is a better choice.
For example:
session_id = f"conversation-{uuid.uuid4()}"
The same ID should then be used for each turn belonging to that conversation.
Sending the First Conversation Turn
Suppose we start with:
messages = [
{
"role": "system",
"content": "You are a helpful travel assistant."
},
{
"role": "user",
"content": "Plan a three-day trip to London."
}
]
Send the request through the Model Router deployment:
first_response = client.chat.completions.create(
model=deployment,
messages=messages,
extra_body=session_affinity,
)
The router selects an eligible model.
For example:
Selected model: Model A
Affinity decision: initialize
The first successful request establishes the model association for the session.
Sending the Second Turn
Now the user asks:
Add restaurant suggestions and indoor activities.
The application needs to keep the conversation history and use the same session ID.
messages.extend(
[
{
"role": "assistant",
"content": first_response.choices[0].message.content
},
{
"role": "user",
"content": "Add restaurant suggestions and indoor activities."
}
]
)
Then send:
second_response = client.chat.completions.create(
model=deployment,
messages=messages,
extra_body=session_affinity,
)
Because the same session ID is being used, Model Router can attempt to retain the previously associated model.
A successful result could look like:
First turn
Model: Model A
Affinity: initialize
Second turn
Model: Model A
Affinity: retain
This is the behavior you generally want when maintaining model consistency across a conversation.
Understanding initialize, retain, and switch
The response can include session-affinity metadata.
The important decisions are:
Decision | Meaning |
|---|
initialize
| No previous association existed and the initially selected model served the request |
retain
| The previously associated model served the request |
switch
| Another model served the request because eligibility or fallback required it |
For example:
{
"session_affinity": {
"mode": "sticky",
"source": "session_id_payload",
"decision": "retain"
}
}
This tells you that the request used sticky session affinity and the associated model was retained.
A switch decision should not automatically be treated as an error.
It can mean the original model could not handle that particular request and Model Router successfully selected another eligible model.
Checking Which Model Actually Served the Request
Even when using session affinity, it is useful to inspect the response.
The Chat Completions response contains a model field identifying the underlying model that served the request.
For example:
print("Serving model:", response.model)
You can also inspect the routing details when the preview metadata is available:
model_selection_details = getattr(
response,
"model_selection_details",
None
)
if model_selection_details:
router_details = model_selection_details.get(
"model_router_details",
{}
)
affinity_details = router_details.get(
"session_affinity"
)
if affinity_details:
print(
"Affinity mode:",
affinity_details.get("mode")
)
print(
"Affinity source:",
affinity_details.get("source")
)
print(
"Affinity decision:",
affinity_details.get("decision")
)
This is useful for troubleshooting because you can distinguish:
Model changed because routing selected another model
from:
Model changed because session affinity had to switch
Session IDs Should Be Managed by Your Application
Model Router does not need to know what your application calls a conversation.
Your application can maintain something like:
Conversation
ID: 8f4d...
User
|
+-- Turn 1
+-- Turn 2
+-- Turn 3
+-- Turn 4
The application maps that conversation to:
session_id = 8f4d...
Every related Chat Completions request uses that identifier.
For another conversation:
session_id = 93ac...
Use a different value.
Do not reuse one session ID for unrelated users or unrelated conversations.
Session Affinity Does Not Store Your Conversation
The session ID is an association mechanism.
It does not mean Model Router becomes your conversation database.
Your application still owns the conversation history.
For example:
messages = [
{"role": "system", "content": "..."},
{"role": "user", "content": "..."},
{"role": "assistant", "content": "..."},
{"role": "user", "content": "..."}
]
Your application continues to send the required history with each Chat Completions request.
Session affinity only helps Model Router associate related requests with an underlying model.
This distinction is important when designing the application architecture.
Session Affinity and Prompt Caching
There is another reason session affinity can be useful.
Consecutive conversation requests often contain overlapping prompt prefixes.
For example:
System instructions
+
Conversation history
+
New user message
If the same model handles consecutive requests, there can be a better opportunity for prompt-cache reuse.
However, session affinity does not guarantee a cache hit.
The router does not use session affinity as a cache-aware model selection mechanism.
Think of it this way:
Session affinity
↓
Try the same eligible model
↓
Potentially better cache reuse
not:
Session affinity
↓
Guaranteed cached prompt
The actual cache behavior depends on the service and request conditions.
The Association Expires
A session association is not permanent.
Microsoft's current documentation states that the model association expires after 30 minutes without a successful create or update.
That means applications should not assume that a session ID permanently maps to one model.
For example:
10:00 Model A
10:10 Model A
10:20 Model A
The association can remain active while successful requests continue updating it.
But after a sufficiently long period without a successful request:
10:20 Last successful request
11:00 New request
the application may need to establish a new association.
This is another reason not to treat session affinity as a permanent model lock.
What Happens During a Model Failure?
Imagine a conversation is associated with:
Model A
A later request arrives.
Model Router tries Model A, but the request gets a retryable failure:
Model A -> 429
The router can then try another eligible model:
Model B -> 200
The response can report:
Affinity decision: switch
The routing trace can show both attempts.
Conceptually:
Session
|
v
Associated Model A
|
| 429
v
Fallback Model B
|
| 200
v
Response
This is a useful design because session consistency should not come at the expense of availability.
When Should You Use Session Affinity?
Session affinity makes the most sense when conversation-level model consistency provides a real benefit.
Long-Running Assistants
A user may interact with an assistant over many turns.
Keeping the same model when possible can make the behavior more predictable.
Coding Assistants
A coding conversation may contain:
Architecture discussion
↓
Code generation
↓
Bug analysis
↓
Refactoring
↓
Testing
Keeping the same eligible model can be useful for maintaining consistent behavior across the conversation.
Customer Support
Support applications often have a long conversation history.
Switching models on every turn is not necessarily bad, but some teams may prefer a more consistent serving model.
Planning Applications
Travel, project planning, research, and similar workflows can have many related turns.
Session affinity can help keep the serving model stable while still allowing fallback when necessary.
When You May Not Need It
Do not enable session affinity just because it exists.
For independent requests such as:
Classify this email.
Summarize this document.
Translate this sentence.
Extract these fields.
there may be little benefit.
If each request stands alone, independent routing can be exactly what you want.
The router can choose the most suitable model for each request.
In that situation:
Request 1 -> Model A
Request 2 -> Model C
Request 3 -> Model B
may be perfectly reasonable.
Session Affinity vs Fixed Model Deployment
There are two different strategies.
Fixed Model
You explicitly choose:
Model A
Every request goes to Model A unless your application changes the deployment.
This gives strong predictability.
Model Router With Session Affinity
You choose:
Model Router
and provide a session ID.
The router tries to keep the conversation on the same eligible model, but it can switch when necessary.
The comparison looks like this:
Feature | Fixed Model | Model Router + Session Affinity |
|---|
Model selection | Application controlled | Router controlled |
Same model across turns | Strongly predictable | Best effort |
Automatic model selection | No | Yes |
Fallback | Application/service dependent | Built into router behavior |
Cost optimization | Manual | Router can optimize |
Flexibility | Lower | Higher |
Model availability handling | Manual | Router can switch when required |
If you absolutely require one specific model for every request, use a direct model deployment instead.
If you want automated routing while improving consistency across a conversation, session affinity is a better fit.
You Can Disable Affinity for a Request
There are cases where you may want normal routing for one particular request.
The session-affinity mode can be set to:
none
For example:
normal_routing = {
"routing_config": {
"session_affinity": {
"mode": "none"
}
}
}
That request uses normal routing and does not read or update the model association.
This can be useful when a specific operation is intentionally independent of the existing conversation routing behavior.
Session Affinity Through a Request Header
Instead of putting the session ID in the request body, the application can provide it through:
x-ms-session-id
The application-owned identifier can therefore be supplied in either location.
If both a valid body identifier and a valid header identifier are present, the body value takes precedence.
This can be useful if your application infrastructure already manages session identifiers at the HTTP layer.
For example:
HTTP Request
|
+-- x-ms-session-id: conversation-123
|
+-- Chat Completions body
Keep the identifier opaque and avoid putting sensitive information into it.
What If the Session ID Is Invalid?
Session IDs have validation requirements.
The identifier must contain between 1 and 256 Unicode code points and include at least one non-whitespace character.
If the body contains an invalid identifier but the header contains a valid identifier, Model Router can use the valid header value.
If neither is valid, Chat Completions falls back to normal routing.
This is another reason to generate session IDs rather than constructing them from arbitrary user input.
Monitoring Session Affinity
When working with a routing system, logging only the final response is not enough.
You should ideally capture:
Conversation ID
Session ID reference
Serving model
Routing mode
Affinity decision
Request latency
Fallback information
HTTP status
Do not log the actual sensitive contents of the session ID if your design includes sensitive values. Better yet, make the session ID opaque from the beginning.
For preview routing metadata, Model Router can expose details such as:
model
routing mode
routing trace
attempted models
HTTP status
latency
session affinity decision
The exact metadata is preview functionality, so applications should handle missing fields gracefully.
A Better Diagnostic Pattern
A production application should not assume that every response contains complete routing metadata.
Use defensive code:
model_selection_details = getattr(
response,
"model_selection_details",
None
)
if not model_selection_details:
print("Routing metadata unavailable")
else:
router_details = model_selection_details.get(
"model_router_details",
{}
)
affinity = router_details.get("session_affinity")
if affinity:
print("Affinity:", affinity)
This matters because session-affinity lookup or persistence can fail without causing inference itself to fail.
In that situation, Model Router can continue with normal routing and the complete affinity object may not be present.
Do Not Treat switch as a Failure
Suppose you receive:
Affinity decision: switch
That does not necessarily mean the request failed.
It can mean:
Associated model
↓
Not eligible or unavailable
↓
Fallback model
↓
Successful response
The request can still be successful.
The important question is:
Why did the router switch?
For that, inspect the routing trace and the top-level model field when the preview metadata is available.
Model Router Still Needs Evaluation
Using Model Router does not remove the need for application-level evaluation.
Before moving a production workload to Model Router, compare it with your current baseline.
At minimum, measure:
Quality
Cost
Latency
Use prompts that represent the real workload.
For example:
100+ representative prompts
can give you a much better signal than a handful of manually selected examples.
Also separate different workload categories.
For example:
Coding
Summarization
Classification
Reasoning
Customer support
Extraction
A router that performs well for one category may behave differently for another.
Microsoft's current evaluation guidance recommends comparing representative workloads rather than relying on a generic quality threshold.
Common Mistakes
Treating Session Affinity as a Hard Lock
It is not.
The router can switch models when required.
Generating a New Session ID for Every Request
If every turn gets a different session ID, the router cannot associate those turns as one session.
Use one session ID for one conversation.
Reusing One Session ID for Everyone
That mixes unrelated conversations.
Generate separate identifiers.
Putting Personal Information in the Session ID
Avoid values such as:
[email protected]
or:
customer-987654
Use opaque identifiers instead.
Forgetting Conversation History
Session affinity does not turn Chat Completions into a stateful conversation store.
Your application still needs to send the appropriate conversation history.
Assuming a Cache Hit
Session affinity may improve the opportunity for cache reuse, but it does not guarantee caching.
Ignoring switch
A model switch can be normal fallback behavior.
Investigate the reason before treating it as a bug.
Depending on Preview Metadata Without Fallback Handling
Preview response fields can be absent.
Write code that works even when routing metadata is unavailable.
A Practical Architecture
A typical application can use this structure:
User
|
v
Application API
|
+---- Conversation ID
|
+---- Session ID
|
v
Foundry Model Router
|
+-------+-------+
| | |
v v v
Model A Model B Model C
|
v
Response
|
v
Application stores
conversation history
The application owns:
Conversation state
Session ID
Authentication
Authorization
Business logic
Model Router owns:
Per-request model selection
Routing policy
Eligible model selection
Fallback behavior
That separation keeps the architecture clean.
A Complete Minimal Example
Here is a simplified Python example:
import os
import uuid
from openai import AzureOpenAI
endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
api_key = os.environ["AZURE_OPENAI_API_KEY"]
deployment = os.environ["MODEL_ROUTER_DEPLOYMENT"]
client = AzureOpenAI(
azure_endpoint=endpoint,
api_key=api_key,
api_version="2024-10-21",
default_headers={
"Foundry-Features": "ModelRouterControls=V1Preview"
},
)
session_id = f"conversation-{uuid.uuid4()}"
session_affinity = {
"routing_config": {
"session_affinity": {
"mode": "sticky",
"session_id": session_id,
}
}
}
messages = [
{
"role": "system",
"content": "You are a helpful travel assistant."
},
{
"role": "user",
"content": "Plan a three-day trip to London."
}
]
first_response = client.chat.completions.create(
model=deployment,
messages=messages,
extra_body=session_affinity,
)
messages.extend(
[
{
"role": "assistant",
"content": first_response.choices[0].message.content,
},
{
"role": "user",
"content": "Add restaurant suggestions and indoor activities.",
},
]
)
second_response = client.chat.completions.create(
model=deployment,
messages=messages,
extra_body=session_affinity,
)
print("First model:", first_response.model)
print("Second model:", second_response.model)
In a real application, you would also add error handling, secure credential management, logging, and appropriate conversation storage.
Summary
Microsoft Foundry Model Router can automatically choose an underlying AI model for each request, helping applications balance quality, cost, and performance.
For multi-turn Chat Completions applications, the new session-affinity preview adds another option. By sending the same application-owned session ID with related requests, the router can try to keep those requests on the same eligible model.
The first successful request normally establishes the association. Later requests can report retain when the same model is used or switch when another model is required.
The key point is that session affinity is best-effort, not a permanent model lock. If the associated model is unavailable or no longer eligible, Model Router can fall back to another model.
For developers, the safest design is simple: generate an opaque session ID for each conversation, reuse it for related turns, keep the conversation history in your own application, monitor the actual serving model, and handle model switches normally.
Join the conversation! Your thoughts help the community grow.