Azure Cosmos DB Change Feed: How to Balance Work Across Multiple Workers

Applications often need to react when data changes in a database. For example, an e-commerce application might need to update a search index after an order is created, while an analytics system may need to process new events as they arrive.

Azure Cosmos DB provides the Change Feed to help applications consume changes made to items in a container. Instead of repeatedly querying the database to discover new records, a consumer can process changes as they become available.

The challenge becomes more complex when an application needs to process a large volume of changes. A single worker may not keep up with incoming data, so multiple workers are often required.

Adding workers alone does not guarantee better performance. The application must distribute work across workers, coordinate progress, handle failures, and prevent duplicate processing from causing incorrect results.

This article explains how the Azure Cosmos DB Change Feed works, how to use the Change Feed Processor, and how to balance processing across multiple workers.

Understanding the Azure Cosmos DB Change Feed

The Change Feed provides an ordered stream of changes to items in a Cosmos DB container. Applications can use it to build event-driven processing pipelines without repeatedly scanning the source container.

Consider an order management application.

When a customer places an order, the application writes an order document to Cosmos DB. A background processor can consume the change and perform tasks such as:

Updating an order analytics system.
Sending an order confirmation through a separate service.
Updating a search index.
Triggering downstream business workflows.
Maintaining a read model for reporting.

The Change Feed separates the database write operation from downstream processing. The application that creates the order does not need to execute every secondary operation synchronously.

Change Feed Processing Options

Azure Cosmos DB supports different approaches to consuming the Change Feed. The appropriate choice depends on the application's architecture and operational requirements.

Approach	Best suited for	Main consideration
Change Feed Processor	Long-running applications that need distributed processing	Requires a lease container to coordinate progress
Azure Functions with a Cosmos DB trigger	Event-driven applications using Azure Functions	Scaling and checkpoint management are handled through the trigger integration
Direct Change Feed APIs	Applications requiring custom polling and checkpoint logic	The application must manage progress and coordination

For most continuously running .NET applications that need multiple workers, the Change Feed Processor is a practical starting point.

How the Change Feed Processor Distributes Work

The Change Feed Processor is designed to distribute processing across multiple instances of an application.

It uses a lease container to coordinate which worker processes each assigned range of the source container's feed.

A lease stores coordination information and progress for a feed range. The processor uses leases to manage ownership and checkpointing as workers start, stop, or scale.

For example, suppose an order container receives thousands of changes per minute.

Initially, one worker may process the available feed ranges. As the workload increases, additional worker instances can join the processing group. The processor can redistribute leases so that multiple workers process different ranges concurrently.

The basic architecture looks like this:

                Azure Cosmos DB
                Orders Container
                       |
                 Change Feed
                       |
             Change Feed Processor
                       |
          +------------+------------+
          |            |            |
       Worker A      Worker B      Worker C
          |            |            |
          v            v            v
       Orders       Analytics     Search Index
       Service       Service       Service
                       |
                 Lease Container
              (Ownership and Progress)

Each worker processes its assigned feed ranges. The lease container coordinates ownership and stores progress information.

The processor manages lease distribution, but it does not automatically make every downstream operation safe to repeat. That responsibility belongs to the application.

Why More Workers Do Not Always Improve Performance

Adding workers increases available processing capacity only when the workload can be distributed across additional feed ranges.

Several factors influence scaling:

The number of available feed ranges.
The rate at which changes arrive.
The time required to process each change.
The throughput available to the source container.
The capacity of downstream services.
The amount of contention between workers.

If the source workload has limited parallelism, adding more instances may provide little improvement. Similarly, if a downstream API is already saturated, increasing the number of consumers can make the overall system less reliable.

Scaling decisions should therefore be based on observed processing lag and resource usage rather than worker count alone.

Setting Up the Change Feed Processor in .NET

The following example demonstrates the main components of a Change Feed Processor application using the Azure Cosmos DB .NET SDK.

The example assumes that the source and lease containers already exist and that the application has suitable permissions.

Install the Required Package

Add the Cosmos DB SDK package to your .NET project:

dotnet add package Microsoft.Azure.Cosmos

The package provides the client and Change Feed Processor APIs.

Create the Cosmos DB Client

The application needs a Cosmos DB client, a reference to the source container, and a reference to the lease container.

using Microsoft.Azure.Cosmos;

string endpoint = Environment.GetEnvironmentVariable(
    "COSMOS_ENDPOINT")
    ?? throw new InvalidOperationException(
        "COSMOS_ENDPOINT is not configured.");

string key = Environment.GetEnvironmentVariable(
    "COSMOS_KEY")
    ?? throw new InvalidOperationException(
        "COSMOS_KEY is not configured.");

CosmosClient client = new CosmosClient(endpoint, key);

Container sourceContainer = client.GetContainer(
    "OrderDatabase",
    "Orders");

Container leaseContainer = client.GetContainer(
    "OrderDatabase",
    "ChangeFeedLeases");

The endpoint and key are loaded from environment variables rather than being hard-coded in the source file.

For production deployments, prefer Microsoft Entra ID and managed identity where supported by the application's hosting environment. This reduces the need to manage long-lived account keys.

The lease container must be configured with an appropriate partition key for the chosen Change Feed Processor setup. Follow the SDK's requirements when creating it.

Implement a Change Handler

The processor calls a delegate when it receives a batch of changes.

The following example logs order IDs and demonstrates where application processing belongs:

using Microsoft.Azure.Cosmos;
using System.Text.Json;

async Task HandleChanges(
    IReadOnlyCollection<JsonElement> changes,
    CancellationToken cancellationToken)
{
    foreach (JsonElement change in changes)
    {
        cancellationToken.ThrowIfCancellationRequested();

        if (change.TryGetProperty("id", out JsonElement id))
        {
            Console.WriteLine($"Processing order: {id.GetString()}");
        }

        // Perform the required business operation here.
        // For example, update a read model or publish an event.
    }

    await Task.CompletedTask;
}

Using JsonElement allows the handler to inspect JSON documents without requiring a dedicated order model.

In a real application, you may prefer a strongly typed model. If you use one, ensure its property names and serialization settings match the documents stored in Cosmos DB.

The handler should process the entire batch successfully before returning. If it throws an exception, the processor can retry the batch, depending on its checkpoint and error-handling behavior.

Build and Start the Processor

The following example creates and starts the processor:

using Microsoft.Azure.Cosmos;
using System.Text.Json;

ChangeFeedProcessor processor = sourceContainer
    .GetChangeFeedProcessorBuilder<JsonElement>(
        processorName: "OrderProcessor",
        onChangesDelegate: HandleChanges)
    .WithInstanceName(
        Environment.MachineName + "-" + Guid.NewGuid())
    .WithLeaseContainer(leaseContainer)
    .Build();

await processor.StartAsync();

Console.WriteLine("Change Feed Processor started.");

Keep the processor running for as long as the application needs to consume changes. In a hosted .NET application, manage its lifetime through the host's startup and shutdown mechanisms.

For example, a BackgroundService can start the processor when the application starts and stop it when the host shuts down. The exact implementation depends on the application's hosting model.

The important point is that multiple application instances should use the same logical processor name and lease container when they are intended to cooperate as one processing group.

How to Balance Work Across Multiple Workers

Once the processor is running, the next task is to ensure that the available workers can process changes at an appropriate rate.

Use a Shared Lease Container

All instances in the same processing group should use the same lease container and processor name.

For example:

Processor name: OrderProcessor

Worker A ──┐
Worker B ──┼── Shared Lease Container
Worker C ──┘

The shared lease container allows the processor to coordinate ownership across instances.

Using different processor names creates separate processing groups. Each group can independently consume the feed, which may be useful when different applications need the same changes. However, it can also increase processing and downstream workload.

Do not create a separate processor name for every worker if those workers are supposed to share the workload.

Scale Based on Processing Lag

Processing lag describes how far behind the consumer is relative to incoming changes.

Suppose the source container receives 10,000 changes per minute, while the application processes only 7,000. The backlog will grow unless the incoming rate decreases or processing capacity improves.

Before adding workers, investigate the cause of the lag.

Useful measurements include:

Change Feed processing delay.
Batch processing duration.
Number of failures and retries.
CPU and memory utilization.
Cosmos DB request consumption and throttling.
Downstream API latency and error rates.

If workers are consistently busy and additional feed ranges can be processed concurrently, scaling out may improve throughput.

If workers spend most of their time waiting for a downstream API, optimize that dependency or introduce controlled concurrency before adding more instances.

Avoid Excessive Concurrency

Increasing concurrency can overwhelm downstream systems.

For example, if every worker sends hundreds of requests per second to a service with a lower rate limit, the downstream service may return throttling errors. Those errors can trigger retries, increasing the load even further.

Use bounded concurrency, batching, and retry policies appropriate to the downstream service.

A common design is to let the Change Feed handler validate and enqueue work, while a separate processing component controls concurrency. However, if the handler checkpoints changes after enqueueing, the queue must provide durable delivery and reliable recovery. An in-memory queue alone is not sufficient when losing accepted work is unacceptable.

Handle Duplicate Processing and Failures

Distributed processing systems must account for retries and worker failures.

The Change Feed Processor manages leases and checkpoints, but application-side operations can still execute more than once. For example, a worker may complete a downstream operation and fail before its progress is safely checkpointed. When processing resumes, the same change may be delivered again.

Make Operations Idempotent

An idempotent operation produces the same final result when repeated with the same input.

Suppose an order change updates an analytics record. Instead of incrementing a counter every time the handler sees the event, the application can use the order ID as an idempotency key and ensure that the same order is not applied twice.

A simplified example is:

string idempotencyKey = $"order:{orderId}";

// Pseudocode:
// 1. Check whether this operation has already been applied.
// 2. If it has, skip the duplicate.
// 3. Otherwise, apply the change and record the key
//    using an atomic or transactionally safe operation.

This is illustrative pseudocode, not a complete implementation. The actual deduplication mechanism depends on the downstream database or service.

For operations that must not be duplicated, use a durable idempotency record, a unique constraint, or a transactional pattern appropriate to the destination.

Treat the Handler as a Retryable Unit

A handler should not assume that it will execute exactly once.

If a batch contains multiple changes and the handler fails after processing some of them, retrying the batch can repeat earlier operations. This is another reason to make side effects idempotent.

Avoid swallowing exceptions and returning success when processing has failed. Doing so can allow progress to advance even though required work was not completed.

Also, do not assume that the Change Feed is a substitute for a fully managed message queue with arbitrary per-message acknowledgment and dead-letter semantics. If your application requires those capabilities, design an appropriate durable messaging layer.

Common Mistakes to Avoid

Mistake	Why it causes problems	Better approach
Giving each worker a different processor name	Workers may create independent processing groups instead of sharing work	Use the same processor name for cooperating instances
Sharing the source container but not the lease container	Workers cannot coordinate progress as intended	Configure the shared lease container consistently
Adding workers without measuring lag	Extra instances may not improve throughput	Monitor processing delay, failures, and downstream capacity
Assuming changes are processed exactly once	Retries can repeat side effects	Make downstream operations idempotent
Ignoring downstream throttling	Retries and failures can increase backlog	Use bounded concurrency and appropriate backoff
Using an in-memory queue for critical work	Accepted work can disappear during a process failure	Use durable storage or a reliable messaging service
Swallowing handler exceptions	Failed work may be treated as successfully processed	Surface failures and implement retry-aware handling

Advantages and Disadvantages

Advantages

The Change Feed supports event-driven processing without repeated full-table polling.
The Change Feed Processor coordinates leases across multiple application instances.
Workers can scale out when the workload has enough parallelism.
Checkpointing allows processing to resume after worker restarts.
The pattern supports integrations with analytics systems, search indexes, and downstream services.

Disadvantages

Processing can lag behind incoming changes during traffic spikes.
Multiple workers do not guarantee linear throughput improvements.
Retry behavior requires idempotent application logic.
Lease management, monitoring, and downstream error handling add operational complexity.
Correctness depends on coordinating checkpoints with successful processing.

Summary

Azure Cosmos DB Change Feed provides a practical way to react to database changes without repeatedly querying the source container. The Change Feed Processor adds lease-based coordination so multiple application instances can share the processing workload.

To balance work effectively, use a shared processor name and lease container, measure processing lag, and scale only when additional parallelism can improve throughput. Monitor downstream dependencies because a slow or throttled service can become the real bottleneck.

Most importantly, design handlers for retries. Use durable idempotency mechanisms for operations that must not be duplicated, and avoid acknowledging work before it has been completed or durably accepted.

A reliable Change Feed architecture depends on more than the number of workers. It requires balanced concurrency, coordinated progress, failure recovery, and careful handling of downstream side effects.