Applications that use AI models often send many requests as part of the same user session. If those requests are routed independently, the application may not always get consistent routing behavior.
Session affinity helps keep related requests associated with the same backend routing context for a period of time.
This can be useful for conversational applications, multi-turn interactions, and workloads where keeping requests together improves consistency.
What Is Session Affinity?
Session affinity, sometimes called sticky routing, means requests from the same session are preferentially routed to the same backend resource or routing context.
Without affinity:
User
|
+-- Request 1 --> Backend A
+-- Request 2 --> Backend C
+-- Request 3 --> Backend B
With affinity:
User
|
+-- Request 1 --> Backend A
+-- Request 2 --> Backend A
+-- Request 3 --> Backend A
The exact routing behavior depends on the service configuration and availability.
For Microsoft Foundry Model Router, the important idea is to provide a consistent session identifier when the application needs session-based routing.
Why Session Affinity Matters for AI Applications
Consider a chat application:
User
|
| "Explain Kubernetes"
v
Model Router
|
v
Model
User
|
| "Give me an example"
v
Model Router
|
v
Model
The second request depends on the context of the first request.
Your application should still send the required conversation context with each request, but consistent routing can be useful when the service supports session affinity.
This is especially relevant when an application sends multiple requests that belong to the same logical interaction.
Use a Stable Session Identifier
The session identifier should represent the logical user session.
For example:
const sessionId = crypto.randomUUID();
Store that value for the lifetime of the conversation.
Do not generate a new identifier for every request:
// Avoid this pattern for session affinity
const sessionId = crypto.randomUUID();
await sendRequest(sessionId);
If the identifier changes every time, there is no stable session to associate with.
A better approach is:
const sessionId = getConversationSessionId();
await sendRequest(sessionId);
await sendRequest(sessionId);
await sendRequest(sessionId);
Keep Session IDs Server-Side When Possible
For applications with authenticated users, generate and manage the session identifier on the server.
A simplified example:
const sessions = new Map();
function getSession(userId) {
if (!sessions.has(userId)) {
sessions.set(userId, crypto.randomUUID());
}
return sessions.get(userId);
}
In a production application, do not use an in-memory Map as the session store for a multi-instance service.
Use an appropriate distributed session mechanism when requests can reach multiple application instances.
Session Affinity Is Not Conversation Memory
These concepts are easy to confuse.
Session affinity:
Request
|
v
Routing behavior
Conversation state:
User message
|
v
Application conversation history
|
v
Model request
Session affinity does not replace application-level conversation management.
If your application needs previous messages, store and send the required context according to the API's conversation model.
A Simplified Request Pattern
A client application can maintain a session identifier:
const session = {
id: crypto.randomUUID()
};
async function sendMessage(message) {
return fetch("/api/chat", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
sessionId: session.id,
message
})
});
}
The server can then associate requests with the same logical session.
The exact Microsoft Foundry request configuration should follow the currently supported Model Router API and authentication mechanism used by your application.
Handling Multiple Users
Do not use one global session ID for all users.
Incorrect:
const sessionId = "global-session";
This can incorrectly associate unrelated users with the same session.
Instead:
function createSession() {
return crypto.randomUUID();
}
Create a separate session for each logical conversation or interaction.
What Happens When a Session Expires?
Session affinity should not be treated as permanent.
Applications should be prepared for routing changes caused by:
Join the conversation! Your thoughts help the community grow.