Applications that use AI models often send many requests as part of the same user session. If those requests are routed independently, the application may not always get consistent routing behavior.

Session affinity helps keep related requests associated with the same backend routing context for a period of time.

This can be useful for conversational applications, multi-turn interactions, and workloads where keeping requests together improves consistency.

What Is Session Affinity?

Session affinity, sometimes called sticky routing, means requests from the same session are preferentially routed to the same backend resource or routing context.

Without affinity:

User
 |
 +-- Request 1 --> Backend A
 +-- Request 2 --> Backend C
 +-- Request 3 --> Backend B

With affinity:

User
 |
 +-- Request 1 --> Backend A
 +-- Request 2 --> Backend A
 +-- Request 3 --> Backend A

The exact routing behavior depends on the service configuration and availability.

For Microsoft Foundry Model Router, the important idea is to provide a consistent session identifier when the application needs session-based routing.

Why Session Affinity Matters for AI Applications

Consider a chat application:

User
 |
 | "Explain Kubernetes"
 v
Model Router
 |
 v
Model

User
 |
 | "Give me an example"
 v
Model Router
 |
 v
Model

The second request depends on the context of the first request.

Your application should still send the required conversation context with each request, but consistent routing can be useful when the service supports session affinity.

This is especially relevant when an application sends multiple requests that belong to the same logical interaction.

Use a Stable Session Identifier

The session identifier should represent the logical user session.

For example:

const sessionId = crypto.randomUUID();

Store that value for the lifetime of the conversation.

Do not generate a new identifier for every request:

// Avoid this pattern for session affinity
const sessionId = crypto.randomUUID();

await sendRequest(sessionId);

If the identifier changes every time, there is no stable session to associate with.

A better approach is:

const sessionId = getConversationSessionId();

await sendRequest(sessionId);
await sendRequest(sessionId);
await sendRequest(sessionId);

Keep Session IDs Server-Side When Possible

For applications with authenticated users, generate and manage the session identifier on the server.

A simplified example:

const sessions = new Map();

function getSession(userId) {
  if (!sessions.has(userId)) {
    sessions.set(userId, crypto.randomUUID());
  }

  return sessions.get(userId);
}

In a production application, do not use an in-memory Map as the session store for a multi-instance service.

Use an appropriate distributed session mechanism when requests can reach multiple application instances.

Session Affinity Is Not Conversation Memory

These concepts are easy to confuse.

Session affinity:

Request
   |
   v
Routing behavior

Conversation state:

User message
   |
   v
Application conversation history
   |
   v
Model request

Session affinity does not replace application-level conversation management.

If your application needs previous messages, store and send the required context according to the API's conversation model.

A Simplified Request Pattern

A client application can maintain a session identifier:

const session = {
  id: crypto.randomUUID()
};

async function sendMessage(message) {
  return fetch("/api/chat", {
    method: "POST",
    headers: {
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      sessionId: session.id,
      message
    })
  });
}

The server can then associate requests with the same logical session.

The exact Microsoft Foundry request configuration should follow the currently supported Model Router API and authentication mechanism used by your application.

Handling Multiple Users

Do not use one global session ID for all users.

Incorrect:

const sessionId = "global-session";

This can incorrectly associate unrelated users with the same session.

Instead:

function createSession() {
  return crypto.randomUUID();
}

Create a separate session for each logical conversation or interaction.

What Happens When a Session Expires?

Session affinity should not be treated as permanent.

Applications should be prepared for routing changes caused by:

The application should therefore remain correct even when two requests are not ultimately handled by the same backend context.

This is an important production design principle.

Session Affinity and Scaling

Consider an application with multiple application servers:

                    +--> App Instance A
User --> Load Balancer
                    +--> App Instance B
                    +--> App Instance C

If session state exists only in memory on Instance A, routing the next request to Instance B can cause problems.

For scalable systems, keep important session state in a shared data store or use a stateless application design.

Session affinity at one layer should not be confused with application session storage at another layer.

Security Considerations

A session identifier should not contain sensitive information.

Avoid values such as:

[email protected]
customer-account-number
internal-user-id-with-secrets

Prefer an opaque identifier:

550e8400-e29b-41d4-a716-446655440000

Also validate session ownership on the server.

A user should not be able to simply replace another user's session identifier and access their conversation state.

Logging Session Information Safely

Session IDs can be useful for troubleshooting:

console.log({
  requestId,
  sessionId
});

However, logs should not contain sensitive conversation content or credentials.

It is often better to use a separate request ID:

console.log({
  requestId,
  sessionId,
  status: "completed"
});

This makes tracing easier without logging the complete AI request.

Common Mistakes

Creating a New Session for Every Request

This defeats the purpose of session-based routing.

Treating Affinity as Guaranteed Permanence

Routing systems can change because of availability and infrastructure conditions.

Storing Sessions Only in Application Memory

This becomes problematic when the application scales horizontally.

Using Sensitive Information as a Session ID

Use an opaque identifier instead.

Assuming Affinity Replaces Conversation State

Your application still needs to manage the context required by the model.

Troubleshooting

Requests Do Not Stay Together

Check that the same logical session identifier is being sent consistently.

Also verify that the current Model Router configuration supports the session behavior you are expecting.

Users Share Unexpected State

Check whether your application accidentally uses a global or reused session identifier.

Session State Disappears After Scaling

Review where session data is stored. An in-memory store is local to a single application instance.

Behavior Changes During Failover

Design the application so that correctness does not depend entirely on one backend instance remaining available.

Best Practices

  1. Generate a unique session identifier for each logical conversation.

  2. Keep the identifier stable for that session.

  3. Manage session state server-side when appropriate.

  4. Do not place sensitive information in session IDs.

  5. Keep conversation state separate from routing behavior.

  6. Use distributed storage when session data must survive multiple application instances.

  7. Expect routing changes during failures or infrastructure events.

  8. Log request and session identifiers carefully without exposing sensitive data.

  9. Verify the current Model Router API behavior before deploying configuration changes.

Advantages and Considerations

Area

Advantage

Consideration

Routing

Keeps related requests associated

Affinity should not be treated as permanent

Conversations

Useful for multi-turn workloads

Application still manages conversation state

Debugging

Stable IDs make request tracing easier

Avoid logging sensitive information

Scaling

Can support predictable session behavior

Shared session storage may be required

Reliability

Can improve consistency in supported scenarios

Applications should tolerate routing changes

Summary

Session affinity can be useful when building applications that send multiple related requests through Microsoft Foundry Model Router.

The key is to maintain a stable, unique session identifier while keeping routing behavior separate from application conversation state. The application should remain correct even if routing changes because of expiration, scaling, failover, or service conditions.

For production systems, focus on three things: stable session identification, secure session management, and a design that does not depend on permanent backend affinity.