Let's Design Better A .NET BackgroundService with Distributed Systems Principles

Your .NET BackgroundService is running. The pod is healthy. CPU and memory look normal. There are no obvious errors.

But the queue keeps growing. So where is the problem?

In this article, we’ll follow that problem using a simple (DLCP) approach.

Detect → Locate → Correct → Prevent

Detect: identify that the system is falling behind.

Locate: find where processing capacity is being lost.

Correct: fix the bottleneck and protect the worker from slow dependencies.

Prevent: add the right limits, metrics, and safeguards so the same failure is caught before it becomes an incident.

We won’t start with the code and guess what went wrong. We’ll start with the symptom and work backwards through the system.

At 10:15 AM, everything looks normal.

The health endpoint returns 200. Then someone notices the queue.

Queue depth: 2,400

Thirty minutes later:

Queue depth: 11,800

Another fifteen minutes:

Queue depth: 52,381

The BackgroundService is still running. So why aren’t the messages being processed?

This is where these incidents get interesting.

A background worker can be perfectly alive as a process and still be completely useless as a message processor.

Start with four questions: Detect. Locate. Correct. Prevent.

That’s the path we’ll follow.

Detect

The first mistake is looking at the wrong signal. For a normal API, we might start with:

HTTP 5xx
CPU
Memory
Pod status
Request latency

Those are useful. But a message processor has another metric that matters more:

Is the queue actually moving? Suppose we see:

Queue depth:             52,381
Messages processed/sec:  18
Oldest message age:      24 minutes
Worker pods:             4
Pod status:              Running
CPU:                     12%
Memory:                  41%

Now the problem is obvious. The application is alive. The system isn’t making enough progress. That’s an important distinction. A process can be healthy while the business operation it performs is unhealthy.

For a background worker, I want to know at least:

Queue depth, Oldest message age, Messages processed per second, Processing duration, In-flight messages, Failure rate, Retry count, Last successful processing time

The queue depth tells us there is a problem.

The oldest message tells us how long that problem has existed.

Processing rate tells us whether we’re catching up or falling further behind.

Those three metrics alone can tell a very different story.

Locate

Now we know the worker isn’t keeping up. The next question is:

Where is the time going? Let’s start with the architecture.

The worker itself is simple. A simplified implementation might look like this:

public sealed class OrderWorker(
    IMessageQueue queue,
    IOrderProcessor processor) : BackgroundService
{
    protected override async Task ExecuteAsync(CancellationToken stoppingToken)
    {
        while (!stoppingToken.IsCancellationRequested)
        {
            var message = await queue.ReceiveAsync(stoppingToken);
            await processor.ProcessAsync(message, stoppingToken);
        }
    }
}

The processor does two things:

public async Task ProcessAsync(
  OrderMessage message, 
  CancellationToken cancellationToken)
{
    await _paymentClient.ChargeAsync(message.OrderId, cancellationToken);
    await _orderRepository.MarkAsPaidAsync(message.OrderId, cancellationToken);
}

Nothing obviously wrong. So we follow the request.

The payment call normally takes:

200 ms

During the incident:

30 sec
45 sec
60 sec
90 sec

Now we have something. The worker isn’t spending its time processing orders. It’s spending its time waiting for the payment service.

Locate the Bottleneck

This is where concurrency matters. Concurrency simply means doing multiple operations at the same time instead of waiting for one to finish before starting the next one.

Suppose the worker has 10 messages in flight, meaning it can process up to 10 messages at the same time.

On a normal day, the payment API responds in around 200 ms.

If all 10 slots process messages concurrently:

Concurrency:       10
Processing time:   200 ms

All 10 messages take roughly 200 ms to complete:

 10 messages
───────────── = 50 messages/second
   200 ms

So the worker can theoretically process around 50 messages per second, assuming the queue, database, and other dependencies can keep up.

If those 10 messages were processed sequentially, the calculation would be different:

10 messages × 200 ms = 2,000 ms = 2 seconds

That would give us:

10 messages ÷ 2 seconds = 5 messages/second

That’s why concurrency matters. We’re not waiting for one message to finish before starting the next one.

Then something changes.

The payment service starts having problems. Requests that normally take around 200 ms now take as long as 60 seconds. Those same 10 concurrent slots now look like this:

Worker
│
├── Order 101 → Payment API → waiting
├── Order 102 → Payment API → waiting
├── Order 103 → Payment API → waiting
├── Order 104 → Payment API → waiting
├── ...
└── Order 110 → Payment API → waiting

All 10 slots are occupied for roughly 60 seconds. The throughput becomes:

Concurrency:       10
Processing time:   60 seconds

  10 messages
──────────────
  60 seconds

= 0.17 messages/second

So we went from roughly:

Normal:    ~50 messages/sec
Incident:  ~0.17 messages/sec

That’s a massive drop.

Why Scaling Made It Worse

The obvious response to a growing queue is:

Add more pods. So we go from:

2 pods to 10 pods

Now we potentially have:

10 pods × 20 concurrent operations = 200 in-flight requests

That sounds better. But where are those requests going?

The same payment API.

Suppose the payment service can safely handle 50 concurrent requests. We just sent it 200. Now the payment service gets evem slower. Slower responses keep worker slots occupied longer. More messages accumulate. The queue grows again.

This is an important lesson in distributed systems:

Scaling one component does not necessarily increase the capacity of the system.

Correct

Now that we know where the problem is, we can fix it. The first change is to stop allowing downstream slowness to consume unlimited worker capacity.

1. Bound the concurrency

Instead of letting the consumer process as much work as possible, define a limit.

For example:

var options = new ParallelOptions
{
    MaxDegreeOfParallelism = 20,
    CancellationToken = stoppingToken
};

await Parallel.ForEachAsync(messages, options, ProcessMessageAsync);

Now the worker has an explicit concurrency boundary. But 20 isn't a magic number. It should come from the system.

You need to consider:

2. Put a timeout around slow dependencies

A worker shouldn’t wait forever for a downstream service.

For an HTTP client:

services.AddHttpClient<IPaymentClient, PaymentClient>(client =>
{
    client.BaseAddress = new Uri(configuration["PaymentService:BaseUrl"]!);
    client.Timeout = TimeSpan.FromSeconds(10);
});

Now a request that doesn’t complete within the allowed time releases the worker slot.

But a timeout creates another question: What happens after the timeout?

That’s where retry policy comes in.

3. Don’t retry blindly

Suppose the payment service returns a temporary 503. A retry can make sense. Suppose the message contains an invalid order ID. A retry won’t help.

So failures need classification.

For temporary failures, use controlled retries.

For example:

private static TimeSpan GetRetryDelay(int attempt)
{
    var seconds = Math.Pow(2, attempt);
    return TimeSpan.FromSeconds(seconds);
}

Which gives:

Attempt 1 → 2 sec
Attempt 2 → 4 sec
Attempt 3 → 8 sec
Attempt 4 → 16 sec

In production, add jitter so multiple workers don’t retry at exactly the same time.

When the dependency is struggling, the worker should reduce pressure, not increase it.

4. Stop calling a dependency that is already down

Retries aren’t enough when the dependency is completely unavailable. Imagine thousands of messages doing this:

The worker is wasting capacity on a dependency that isn’t responding.

A circuit breaker changes that.

Instead of allowing every worker to keep discovering that the payment service is down, the system temporarily stops sending requests.

That gives the dependency room to recover.

5. Deal with messages that cannot succeed

Not every message deserves infinite retries.

Imagine:

{
    "orderId": null,
    "amount": "INVALID"
}

Retrying this message 100 times won’t fix it. After the retry limit:

The main queue keeps moving.

The failed message gets a separate path for investigation.

6. Assume messages can be delivered twice

There is another problem that appears in real message systems.

Suppose:

                        Receive message
                              ↓
                        Charge payment
                              ↓
                        Payment succeeds
                              ↓
                        Worker crashes before ACK

The queue doesn’t know the payment succeeded. It may deliver the message again.

Message 123
   │
   ├── Attempt 1 → Payment succeeds
   │
   └── Attempt 2 → Payment succeeds again

Now you’ve potentially charged the customer twice. Message processing should therefore be idempotent.

For example:

public async Task ProcessAsync(
  OrderMessage message, 
  CancellationToken cancellationToken)
{
    if (await _processedMessages.ExistsAsync(message.MessageId, 
                                             cancellationToken))
        return;

    await _paymentClient.ChargeAsync(message.OrderId, cancellationToken);
    await _orderRepository.MarkAsPaidAsync(message.OrderId, cancellationToken);
    await _processedMessages.AddAsync(message.MessageId, cancellationToken);
}

And the database should enforce uniqueness:

CREATE UNIQUE INDEX IX_ProcessedMessages_MessageId
ON ProcessedMessages(MessageId);

The corrected architecture

The worker now has boundaries around the things that can hurt it.

Failure
   │
   ├── Temporary → Retry
   │
   ├── Permanent → DLQ
   │
   └── Repeated → DLQ

Now it has clearer boundaries.

Prevent

Fixing the incident is only half the job. The next question is:

How do we know this is happening before customers notice it?

This is where monitoring changes. A generic health check might say:

Pod: Running
Health: OK

That’s not enough. Now we know, for a message processor, monitor progress. A useful dashboard might would be:

Now we can answer something much more useful than:

Is the pod alive?

We can answer:

Is the system making progress?

The Architecture We Actually Want

A resilient message processor isn’t just:

Queue → BackgroundService → Database

It is closer to:

                 ┌──────────────────────────────┐
                 │        Observability         │
                 │                              │
                 │ Queue depth                  │
                 │ Processing rate              │
                 │ Latency                      │
                 │ Failure rate                 │
                 │ Retry rate                   │
                 │ Oldest message               │
                 │ Last successful processing   │
                 └──────────────────────────────┘

Every part has a job.

The queue absorbs bursts.
The worker controls concurrency.
Timeouts prevent indefinite waits.
Retries handle temporary failures.
Circuit breakers protect unhealthy dependencies.
Idempotency protects against duplicate delivery.
The DLQ prevents poison messages from blocking the system.
Metrics tell us whether the system is actually moving.

The Architectural Lesson

The interesting part of this problem is that there was no single broken line of code.

The worker was doing what it was designed to do.
The payment service was doing what it could.
Kubernetes was reporting the truth: the pods were running.
And the queue was doing its job too.
The failure appeared between those components.

That is where distributed-system problems usually live.

A BackgroundService is just a loop:

while (!stoppingToken.IsCancellationRequested)
{
    // Do some work
}

The hard part is everything around that loop.

How much work can it take?
What happens when a dependency slows down?
How long can it wait?
How many times should it retry?
What happens when the same message arrives twice?
What happens to a message that can never succeed?

And most importantly:

How do we know the worker is making progress?

A running worker is not necessarily a healthy worker.

A healthy pod is not necessarily a healthy message-processing system.

The metric that matters is not whether the loop is alive.

It’s whether the queue is moving.

Cheers,

Rikam Palkar

mvp.microsoft.com/en-US/mvp