In an ASP.NET Core application, not every operation finishes while the HTTP request is still active. A user might submit an order, upload a document, request a report, or trigger an email notification. The API receives the request, validates the input, and then schedules additional work to run in the background.
This approach keeps HTTP responses fast and prevents users from waiting for operations that do not need to complete immediately. However, it introduces an observability problem that is easy to miss: once the request finishes, how do you connect the background operation to the original HTTP request?
Consider an order management application. A customer submits an order through an API. The application saves the order and places a message into a background queue. A hosted service later picks up that message, generates an invoice, updates the order status, and sends a notification.
Suppose invoice generation fails several seconds after the API has already returned a successful response. When the development team investigates the failure, the background worker logs show an exception, but the original request logs show that the order was created successfully. Without a shared tracing context, developers may have to search manually using the order ID and timestamps to understand what happened.
The problem becomes more complicated when multiple customers submit orders simultaneously and background workers process several messages concurrently.
.NET provides distributed tracing through Activity, ActivitySource, and ActivityContext. These APIs make it possible to preserve the relationship between an incoming HTTP request and work that continues after the request has completed.
The important part is understanding how tracing context flows across asynchronous boundaries, especially when using queues and hosted services.
Why Background Processing Breaks the Normal Request Flow
ASP.NET Core automatically creates an activity for an incoming HTTP request when the framework's diagnostic instrumentation is active. This activity represents the request and provides information such as its trace ID, span ID, and duration.
During normal asynchronous execution, the current activity flows across awaited operations. For example, when a controller calls an application service and that service calls a repository, the activities can remain part of the same trace.
Background processing is different.
When an application places a message into a queue, it is effectively separating the work from the current request execution. The background worker may process the message later, on another thread, after the HTTP request has completed.
The worker should not depend on the original HTTP execution context still being available. It needs enough information to establish a tracing relationship with the request that scheduled the work.
Consider the following flow:
HTTP Request
|
v
Order API
|
+------ Save Order
|
+------ Add Work Item to Queue
|
v
Return HTTP Response
|
| Request Activity Ends
|
v
Background Worker
|
+------ Generate Invoice
|
+------ Update Order
|
+------ Send Notification
The background worker performs valid application work, but the original request activity may already have ended.
If the worker starts a completely new activity without the original context, the tracing system treats it as an unrelated operation. The logs may contain the correct order ID, but the trace visualization will not show that the background processing originated from the customer's request.
The solution is to capture the trace context before placing the work item into the queue and explicitly use that context when the background worker starts processing it.
Understand ActivityContext and ActivitySource
Before implementing the queue, it helps to understand two important .NET tracing types.
ActivitySource creates activities for application operations. Think of it as the tracing source associated with a component or library. A service can use it to create activities around meaningful operations such as processing an order, generating a report, or sending a notification.
ActivityContext contains the information required to establish a tracing relationship. It includes the trace ID, span ID, trace flags, and trace state.
When an ASP.NET Core request is being processed, the current activity is available through Activity.Current.
The following code captures its context:
using System.Diagnostics;
ActivityContext? parentContext =
Activity.Current?.Context;
This context can be stored with a background work item.
Later, the background worker can use it when creating a new activity:
using var activity = ActivitySource.StartActivity(
"GenerateInvoice",
ActivityKind.Consumer,
parentContext);
The new activity receives its own span ID but retains the trace ID from the original request. This creates a parent-child relationship that tracing platforms can display as part of the same distributed trace.
The distinction between capturing an ActivityContext and keeping an Activity object is important. The context is a lightweight representation of tracing identity. It does not require the original HTTP request or its activity object to remain alive.
Create a Background Queue That Carries Trace Context
For an in-process background queue, .NET's Channel<T> provides an asynchronous producer-consumer mechanism. It allows an API request to enqueue work while a hosted service consumes the items independently.
For this example, define a work item that contains the order ID and the tracing context.
using System.Diagnostics;
public sealed record OrderWorkItem(
int OrderId,
ActivityContext ParentContext);
The OrderId contains the business information required by the worker. ParentContext preserves the relationship with the request that created the work item.
Next, define the queue interface:
public interface IOrderBackgroundQueue
{
ValueTask EnqueueAsync(
OrderWorkItem item,
CancellationToken cancellationToken);
IAsyncEnumerable<OrderWorkItem> ReadAllAsync(
CancellationToken cancellationToken);
}
The implementation uses a bounded channel:
using System.Threading.Channels;
public sealed class OrderBackgroundQueue
: IOrderBackgroundQueue
{
private readonly Channel<OrderWorkItem> _channel;
public OrderBackgroundQueue()
{
_channel = Channel.CreateBounded<OrderWorkItem>(
new BoundedChannelOptions(500)
{
FullMode = BoundedChannelFullMode.Wait,
SingleReader = true,
SingleWriter = false
});
}
public ValueTask EnqueueAsync(
OrderWorkItem item,
CancellationToken cancellationToken)
{
return _channel.Writer.WriteAsync(
item,
cancellationToken);
}
public IAsyncEnumerable<OrderWorkItem> ReadAllAsync(
CancellationToken cancellationToken)
{
return _channel.Reader.ReadAllAsync(
cancellationToken);
}
}
The bounded channel is intentional. An unbounded queue can continue accepting work even when the background worker cannot keep up. During a traffic spike, the application may accumulate more work than it can process, increasing memory consumption.
With a bounded channel, the producer waits when the queue reaches its capacity. The application can then apply backpressure rather than allowing the queue to grow indefinitely.
The capacity of 500 is only an example. A production application should select a limit based on its workload, processing time, and memory requirements.
Capture the Request Context Before Enqueuing Work
The API should capture the current activity context while the original HTTP request is still being processed.
Consider an order controller:
using System.Diagnostics;
using Microsoft.AspNetCore.Mvc;
[ApiController]
[Route("api/orders")]
public class OrdersController : ControllerBase
{
private readonly IOrderBackgroundQueue _queue;
public OrdersController(IOrderBackgroundQueue queue)
{
_queue = queue;
}
[HttpPost("{orderId:int}/invoice")]
public async Task<IActionResult> GenerateInvoice(
int orderId,
CancellationToken cancellationToken)
{
var parentContext =
Activity.Current?.Context ?? default;
var workItem = new OrderWorkItem(
orderId,
parentContext);
await _queue.EnqueueAsync(
workItem,
cancellationToken);
return Accepted(new
{
OrderId = orderId,
Message = "Invoice generation has been scheduled."
});
}
}
The controller captures Activity.Current?.Context before writing the work item to the queue.
This timing matters. Once the HTTP request finishes, its activity may be stopped and Activity.Current may no longer refer to the original request.
The controller then returns HTTP 202 Accepted, indicating that the operation has been accepted for asynchronous processing rather than completed.
The example uses the request cancellation token while writing to the queue. This is appropriate because the enqueue operation is part of the HTTP request. If the client disconnects before the work is accepted, the write can be cancelled.
Once the item has been successfully accepted, the background operation should have its own lifecycle. It should not continue using the original request cancellation token.
Start a New Activity Inside the Background Worker
The background worker is responsible for reading work items and executing them.
ASP.NET Core provides BackgroundService as a convenient base class for implementing long-running hosted services.
The worker should create a dependency injection scope for each work item when it needs scoped services such as repositories or Entity Framework Core database contexts.
using System.Diagnostics;
public sealed class OrderBackgroundWorker
: BackgroundService
{
private readonly IOrderBackgroundQueue _queue;
private readonly IServiceScopeFactory _scopeFactory;
private readonly ILogger<OrderBackgroundWorker> _logger;
private static readonly ActivitySource ActivitySource =
new("OrderApi.Background");
public OrderBackgroundWorker(
IOrderBackgroundQueue queue,
IServiceScopeFactory scopeFactory,
ILogger<OrderBackgroundWorker> logger)
{
_queue = queue;
_scopeFactory = scopeFactory;
_logger = logger;
}
protected override async Task ExecuteAsync(
CancellationToken stoppingToken)
{
await foreach (
var item in _queue.ReadAllAsync(stoppingToken))
{
using var activity = ActivitySource.StartActivity(
"ProcessOrderBackground",
ActivityKind.Consumer,
item.ParentContext);
activity?.SetTag("order.id", item.OrderId);
try
{
using var scope = _scopeFactory.CreateScope();
var processor = scope.ServiceProvider
.GetRequiredService<IOrderProcessor>();
await processor.ProcessAsync(
item.OrderId,
stoppingToken);
_logger.LogInformation(
"Background processing completed for order {OrderId}",
item.OrderId);
}
catch (Exception ex)
{
activity?.SetStatus(
ActivityStatusCode.Error,
ex.Message);
_logger.LogError(
ex,
"Background processing failed for order {OrderId}",
item.OrderId);
}
}
}
}
The important line is the call to StartActivity() with item.ParentContext.
The worker creates a new activity for the background operation while preserving the trace context captured by the controller. This means the background operation has its own span but belongs to the original request's trace.
The ActivityKind.Consumer value is appropriate for work consumed from a queue. It communicates the nature of the operation to tracing systems.
The dependency injection scope is another important detail. A hosted service is typically registered as a singleton, so it should not directly hold a scoped database context for its entire lifetime. Creating a scope for each work item ensures that scoped dependencies have an appropriate lifetime.
The worker uses its own stoppingToken because processing is now controlled by the hosted service lifecycle rather than the original HTTP request.
Register the Queue and Background Worker
The queue must be registered as a singleton so that the controller and background worker use the same channel instance.
The worker is registered as a hosted service:
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddSingleton<
IOrderBackgroundQueue,
OrderBackgroundQueue>();
builder.Services.AddHostedService<OrderBackgroundWorker>();
builder.Services.AddScoped<IOrderProcessor, OrderProcessor>();
builder.Services.AddControllers();
var app = builder.Build();
app.MapControllers();
app.Run();
If the queue is registered separately for the controller and hosted service, each component could receive a different instance. The controller would write to one channel while the worker reads from another, and the queued work would never be processed.
Using a singleton queue avoids this problem for an in-process implementation.
Configure OpenTelemetry to Record the Background Activity
Creating an ActivitySource does not automatically guarantee that its activities will appear in a tracing platform. The telemetry provider must subscribe to the source.
For an ASP.NET Core application, OpenTelemetry can be configured to collect both incoming request activities and custom background activities.
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;
builder.Services.AddOpenTelemetry()
.ConfigureResource(resource =>
resource.AddService("OrderApi"))
.WithTracing(tracing =>
{
tracing
.AddAspNetCoreInstrumentation()
.AddHttpClientInstrumentation()
.AddSource("OrderApi.Background");
});
The AddAspNetCoreInstrumentation() call enables collection of incoming HTTP request activities. AddSource("OrderApi.Background") subscribes to the custom activity source used by the worker.
An exporter must also be configured to send the collected traces to the selected observability platform.
If the custom activity source is not registered, the worker's activity may not be collected even though the code calls StartActivity() correctly.
This is a common reason why developers see HTTP request traces but cannot find their custom background operations.
Understand the Relationship Between the Request and Background Span
After the implementation is running, the tracing platform should be able to display the original request and the background operation under the same trace.
For example:
Trace: 4bf92f3577b34da6a3ce929d0e0e4736
POST /api/orders/10542/invoice
|
+---- SaveOrder
|
+---- EnqueueInvoiceWork
|
+---- ProcessOrderBackground
|
+---- GenerateInvoice
|
+---- UpdateOrder
|
+---- SendNotification
The HTTP request and background operation have different span IDs, but they share the same trace ID.
This makes it possible to investigate questions that are difficult to answer with ordinary application logs:
Which HTTP request scheduled this background operation?
How long did the background processing take?
Did invoice generation or notification delivery fail?
Did the background operation start immediately or remain in the queue?
Are several failed background operations associated with the same incoming request?
There is one important distinction concerning duration. The HTTP request span ends when the API returns its response. The background activity begins when the worker starts processing the item. The time spent waiting in the queue is not automatically represented as part of the background processing duration.
If queue delay is important, record the enqueue timestamp and add a separate queue-wait measurement. This helps distinguish slow processing from a worker that is unable to keep up with incoming work.
What Changes When Background Work Runs in Another Process?
The in-process channel example works when the API and background worker run inside the same application process. Many production systems use a message broker such as RabbitMQ, Azure Service Bus, or Kafka instead.
In that architecture, the tracing context must travel with the message.
A message producer should capture the current trace context and inject it into the message headers using the W3C Trace Context format or the messaging instrumentation provided by the selected framework.
The consumer extracts that context and starts its processing activity using the extracted parent context.
The concept remains the same, but the transport changes:
In-process background queue | External message broker |
|---|
Store ActivityContext in the work item | Propagate trace context through message headers |
Read context from the queued object | Extract context from the incoming message |
Start a consumer activity | Start a consumer activity |
Use the same process's tracing provider | Configure tracing in both producer and consumer |
Do not serialize the entire Activity object into a message. It contains runtime state that is not intended to be transferred between processes. Use the supported trace-context propagation mechanisms instead.
Also, message delivery semantics need separate consideration. A message broker may deliver a message more than once, so background processing should be designed to handle duplicate delivery where necessary. Trace correlation helps diagnose processing, but it does not provide exactly-once execution or transaction guarantees.
Common Mistakes to Avoid
One common mistake is relying on Activity.Current inside the background worker. The worker may execute after the original request has ended, so there is no guarantee that the original activity will still be current. Capture ActivityContext before enqueuing the work.
Another mistake is storing HttpContext or request-scoped services in the queue. These objects belong to the HTTP request lifecycle and should not be used as long-lived background work data. Store the required business information and tracing context instead.
A third mistake is passing HttpContext.RequestAborted to work that should continue after the response has been accepted. Once the request is complete, the background operation should follow its own cancellation and shutdown policy.
Teams should also avoid creating a completely new root trace for every background operation when the intention is to connect it to the original request. Use the captured parent context to preserve the relationship.
Finally, remember that an in-memory channel is not durable. If the application process stops unexpectedly, queued items can be lost. Important business operations should generally use a durable message broker or a persistent queue, along with retry and failure-handling mechanisms.
Summary
ASP.NET Core background processing introduces a gap in request tracing because the HTTP request can finish before the actual work begins. Without explicit context propagation, background operations may appear as unrelated traces, making production troubleshooting difficult.
.NET provides the necessary building blocks through Activity, ActivitySource, and ActivityContext. Capture the current request context before enqueuing work, store it with the work item, and use it as the parent context when the background worker starts a new activity.
For an in-process queue, a bounded Channel<T> and BackgroundService provide a practical implementation. For work that crosses process boundaries, propagate the trace context through message headers and extract it in the consumer.
The result is a connected view of the operation, from the original HTTP request through the background processing stages. Developers can investigate failures using a shared trace instead of trying to reconstruct the workflow from unrelated log messages.
Join the conversation! Your thoughts help the community grow.