Azure Cosmos DB for AI Agent Memory: Building Cross-Session Context

An AI agent can remember a conversation while the current request is running, but that does not mean it has useful memory.

If a user starts a new session tomorrow, the agent should still be able to recall important information such as preferences, previous decisions, unresolved tasks, or facts the user explicitly shared. That requires persistent memory outside the model's current context window.

Azure Cosmos DB for NoSQL is a practical option for this because it can store conversation records as JSON documents and also support semantic retrieval with vector search. This makes it possible to keep raw conversations, summaries, facts, and user-level information in one data layer.

The important part is the memory design. Simply storing every chat message and sending everything back to the model is not a good memory strategy.

Short-Term Context vs Long-Term Memory

An agent usually needs two different types of memory.

Memory type	Purpose	Typical lifetime
Short-term memory	Current conversation and recent turns	Minutes to hours
Thread memory	Summary of a conversation	Days or longer
Semantic memory	Important facts and preferences	Long term
User memory	Information shared across conversations	Long term

For example, suppose a user tells an AI coding assistant:

"I prefer MSTest for my .NET projects."

The current conversation can contain that sentence, but a new session will not automatically contain it.

A memory system can extract the preference and store something similar to:

{
  "id": "fact-9f2c",
  "userId": "user-123",
  "type": "fact",
  "content": "The user prefers MSTest for .NET projects.",
  "sourceThreadId": "thread-456",
  "createdAt": "2026-09-10T08:30:00Z"
}

During a future conversation, the application can retrieve that memory and add it to the agent's context.

The model does not need the complete historical conversation. It needs the relevant part of the history.

A Practical Cosmos DB Memory Model

A simple memory container can use a document structure like this:

{
  "id": "memory-001",
  "userId": "user-123",
  "threadId": "thread-456",
  "type": "fact",
  "content": "The user prefers MSTest for .NET projects.",
  "importance": 0.85,
  "createdAt": "2026-09-10T08:30:00Z",
  "expiresAt": null,
  "embedding": [0.012, -0.083, 0.221]
}

The important fields are:

userId identifies the owner of the memory.
threadId connects the memory to the conversation that produced it.
type separates turns, summaries, facts, and user-level memories.
content contains the information the agent can use.
importance can help decide which memories deserve longer retention.
expiresAt can support temporary memories.
embedding enables semantic retrieval.

A production system does not have to put every type of memory into one container. Separate containers can make sense when access patterns and retention policies differ.

Choosing the Partition Key

Partition-key design matters because memory retrieval normally starts with a user or tenant.

For a single-user application, /userId can be a reasonable starting point:

/userId

For a multi-tenant application, you may need a tenant-aware design:

/tenantId

or a hierarchical strategy that reflects the application's access pattern.

The important question is not "Which property looks unique?" It is:

Which value will the application normally use when reading and writing memory?

If almost every query asks for a user's memories, the partitioning strategy should support that access pattern.

Also watch for hot partitions. A single partition key value used by a large number of requests can become a bottleneck as the application grows.

Storing Conversation Turns

The first step is usually to persist the conversation itself.

A C# model can stay simple:

public sealed class MemoryDocument
{
    public string Id { get; set; } = default!;
    public string UserId { get; set; } = default!;
    public string ThreadId { get; set; } = default!;
    public string Type { get; set; } = default!;
    public string Content { get; set; } = default!;
    public DateTime CreatedAt { get; set; }
    public double? Importance { get; set; }
    public float[]? Embedding { get; set; }
}

Writing a memory document with the .NET SDK looks like this:

var memory = new MemoryDocument
{
    Id = Guid.NewGuid().ToString(),
    UserId = userId,
    ThreadId = threadId,
    Type = "turn",
    Content = userMessage,
    CreatedAt = DateTime.UtcNow
};

await container.CreateItemAsync(
    memory,
    new PartitionKey(memory.UserId));

The application should create one reusable CosmosClient rather than creating a new client for every request.

Do Not Send Every Memory to the Model

This is one of the easiest mistakes to make.

Imagine that a user has accumulated 5,000 memory records. Loading all 5,000 documents into every prompt defeats the purpose of having a memory system.

A better flow is:

User request
     |
     v
Generate search query
     |
     v
Retrieve relevant memories
     |
     +---- Recent conversation
     |
     +---- User facts
     |
     +---- Relevant summaries
     |
     v
Build agent context
     |
     v
Call model

The agent should retrieve only the memories relevant to the current task.

For example, if the user asks:

"Create a new .NET API using my usual testing setup."

The application can search memories related to testing preferences and retrieve:

User prefers MSTest.
User usually uses .NET Web API projects.
Previous projects use integration tests for API endpoints.

That is much more useful than replaying thousands of old messages.

Adding Semantic Memory with Vector Search

Keyword queries work when the wording is predictable. Memory retrieval is often different.

A user may have stored:

I prefer MSTest for my .NET projects.

but later ask:

Use the testing framework I normally choose.

The words are different, but the meaning is similar.

Vector search can help with this type of retrieval.

Azure Cosmos DB for NoSQL supports storing embeddings alongside the original document and querying those vectors with vector search. A memory document can therefore contain both the human-readable information and its embedding.

A conceptual query looks like this:

SELECT TOP 5
    c.id,
    c.content,
    VectorDistance(c.embedding, @queryVector) AS distance
FROM c
WHERE c.userId = @userId
ORDER BY VectorDistance(c.embedding, @queryVector)

The application first generates an embedding for the current request, then uses that vector to find semantically similar memories.

For production workloads, vector indexing and partitioning should be designed together. The vector query is only useful if the underlying data model supports the application's retrieval pattern.

Use Summaries Instead of Keeping Everything

Raw conversation history is useful for auditing and replay, but it is usually too large to send directly to an LLM.

A better approach is to periodically create a summary:

{
  "id": "summary-thread-456",
  "userId": "user-123",
  "threadId": "thread-456",
  "type": "summary",
  "content": "The user is building a .NET application and prefers MSTest. The current task is adding API integration tests.",
  "createdAt": "2026-09-10T09:00:00Z"
}

The summary can replace many individual turns when the agent needs historical context.

The application can still keep the original turns for cases where detailed history is required.

Use the Change Feed for Memory Processing

Memory extraction does not have to happen inside the user's request.

A useful architecture is:

Conversation
    |
    v
Cosmos DB
    |
    v
Change Feed
    |
    v
Background processor
    |
    +--> Create summary
    |
    +--> Extract facts
    |
    +--> Generate embeddings
    |
    v
Memory documents

Azure Cosmos DB change feed provides a persistent stream of document changes that can be processed asynchronously.

This is useful when creating a summary or generating embeddings would make the user's request slower.

For example, the application can save the conversation immediately and let a background process create long-term memories afterward.

This also gives you a cleaner separation between the request path and memory-processing workload.

Handling Conflicting Memories

Memory can become incorrect.

Suppose an agent stores:

User prefers MSTest.

Six months later, the user says:

I have switched to xUnit for new projects.

The system should not blindly create two equally authoritative memories.

A memory record can contain fields such as:

{
  "type": "fact",
  "content": "The user now prefers xUnit for new projects.",
  "status": "active",
  "supersedes": "fact-001"
}

Another option is to mark the old record as superseded.

This matters because memory is persistent. A wrong fact can continue influencing future sessions long after the original conversation is forgotten.

Protect Memory with Optimistic Concurrency

Multiple agent requests may update the same user memory at the same time.

Azure Cosmos DB provides ETags for optimistic concurrency. An update can require the document to still have the ETag that was read previously.

Conceptually:

var requestOptions = new ItemRequestOptions
{
    IfMatchEtag = existingDocument.ETag
};

await container.ReplaceItemAsync(
    updatedDocument,
    updatedDocument.Id,
    new PartitionKey(updatedDocument.UserId),
    requestOptions);

If another operation changed the document first, the update can fail instead of silently overwriting the newer version.

This is especially useful when multiple agent workers can modify the same profile or memory record.

Memory Expiration and Retention

Not every memory should live forever.

A temporary memory might be useful for a few hours:

{
  "type": "temporary",
  "content": "The user is currently debugging the payment API.",
  "expiresAt": "2026-09-12T08:30:00Z"
}

Cosmos DB TTL can be used when the data model requires automatic expiration.

Retention should be part of the memory design, not something added after the system starts accumulating data.

A useful policy might look like this:

Memory	Retention
Raw conversation turns	Limited or policy-based
Thread summaries	Longer retention
User preferences	Until changed or deleted
Temporary task context	Short TTL
Sensitive information	Store only when necessary

The exact retention period depends on the application and its compliance requirements.

Azure Cosmos DB Agent Memory Toolkit

Microsoft also provides an Agent Memory Toolkit for Azure Cosmos DB for NoSQL. It supports memory concepts such as turns, summaries, facts, and user summaries, along with semantic and hybrid retrieval.

The toolkit is currently a preview and uses Python. It can be useful when an application wants a ready-made memory workflow instead of implementing every memory operation manually.

For a C# application, the same architectural ideas can be implemented directly with the Azure Cosmos DB .NET SDK.

That distinction matters. A preview toolkit should not automatically become a production dependency just because it simplifies development.

Common Mistakes

Treating conversation history as memory

Conversation history is raw data. Memory is selected information that the agent can use later.

Saving every message as a permanent fact

Most messages are not worth remembering. Store durable facts selectively.

Using only vector similarity

Semantic similarity can retrieve related information, but structured filters such as user, tenant, memory type, or status are still important.

Ignoring conflicting memories

A memory system needs a way to update, supersede, or invalidate old information.

Putting memory processing in the request path

Embedding generation and summarization can increase response time. Background processing is often a better design.

Choosing the partition key too late

Changing the data model after significant production data has accumulated can be difficult. Decide the main retrieval pattern before creating the container.

Best Practices

Separate short-term context from long-term memory.
Partition data around the application's normal access pattern.
Store raw turns separately from derived facts and summaries when that improves management.
Retrieve only the memories relevant to the current request.
Use vector search when semantic retrieval adds value.
Combine semantic retrieval with metadata filters.
Process summaries and embeddings asynchronously when possible.
Use ETags when multiple workers can update the same memory.
Apply TTL and retention rules to temporary information.
Give users a way to correct or delete stored memories.
Treat sensitive information differently from ordinary preferences.
Monitor RU consumption, latency, throttling, and memory growth.

Advantages and Disadvantages

Advantages	Disadvantages
JSON document model fits conversation data	Memory design still requires application logic
Supports semantic vector search	Vector indexing adds configuration and resource usage
Change feed supports asynchronous processing	Background processing introduces eventual consistency
Scales with partitioned workloads	Poor partition-key choices can hurt performance
ETags support optimistic concurrency	Conflicting memories need explicit application rules
TTL can automate expiration	Long-term memory requires careful retention policies
.NET SDK integrates directly with C# applications	Retrieval quality depends on the memory and embedding strategy

A Practical Architecture for a .NET AI Agent

For a production-oriented .NET application, a reasonable starting architecture is:

                 +----------------+
                 |   AI Agent     |
                 +-------+--------+
                         |
              +----------+----------+
              |                     |
        Recent context       Memory retrieval
                                    |
                                    v
                           +----------------+
                           | Azure Cosmos DB|
                           +-------+--------+
                                   |
                       +-----------+-----------+
                       |                       |
                 JSON memories            Vector search
                       |
                       v
                  Change Feed
                       |
                       v
               Background worker
                 |            |
              Summary       Facts
                 |            |
                 +------> Memory

The agent writes conversation data to Cosmos DB. A background process turns useful conversation content into summaries and facts. When the user starts another session, the application searches those memories and adds only relevant results to the model's context.

That gives the agent something more useful than a giant conversation log: persistent, searchable context that can survive across sessions and be updated when the user's information changes.