OpenTelemetry in .NET: How to Trace Requests Across Microservices

When an application runs as a single service, investigating a slow request is relatively straightforward. Developers can inspect application logs, debug the code, and identify the operation responsible for the delay.

The situation becomes more complicated when an application consists of multiple microservices. A single request might pass through an API gateway, an order service, a payment service, and a database before returning a response. If the request takes several seconds, identifying which component caused the delay can be difficult.

OpenTelemetry helps solve this problem by collecting telemetry from different services and connecting related operations through distributed traces. Developers can follow a request across service boundaries, inspect how much time each operation takes, and identify failures without manually correlating unrelated log files.

In this article, we will implement distributed tracing in an ASP.NET Core application, instrument outgoing HTTP requests, create custom spans, propagate trace context between services, and examine practices that make tracing useful in production.

What Is OpenTelemetry?

OpenTelemetry is an open-source observability framework that provides APIs, instrumentation libraries, and exporters for collecting telemetry from applications.

It supports three commonly used forms of telemetry:

Traces describe the path of an operation through one or more services.
Metrics provide numerical measurements such as request duration, error rates, and active requests.
Logs record individual events and application messages.

These signals complement one another. Metrics can reveal that an API has become slower, traces can identify which service contributes most to the delay, and logs can provide additional information about the failure.

Understanding Distributed Tracing

A distributed trace represents the complete journey of an operation through an application.

For example, consider an online shopping application:

A customer submits an order through the API.
The order service validates the request.
The order service calls the payment service.
The payment service communicates with an external provider.
The order service saves the result and returns a response.

Each operation can be represented by a span. Related spans belong to the same trace and can be inspected together.

A trace contains a trace ID that identifies the overall operation. Each span has its own span ID, and child spans are associated with their parent operations.

This structure helps developers determine whether a delay originated in the order service, the payment service, or an external dependency.

Create an ASP.NET Core Application

Start by creating a new Web API project:

dotnet new webapi -n OrderApi
cd OrderApi

Add the OpenTelemetry packages required for ASP.NET Core request tracing, outgoing HTTP instrumentation, and console-based debugging.

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.Console

The packages serve different purposes:

OpenTelemetry.Extensions.Hosting integrates telemetry configuration with the .NET hosting model.
OpenTelemetry.Instrumentation.AspNetCore collects telemetry for incoming ASP.NET Core requests.
OpenTelemetry.Instrumentation.Http instruments outgoing HTTP requests made through HttpClient.
OpenTelemetry.Exporter.Console writes collected telemetry to the console for development and troubleshooting.

The console exporter is convenient for learning and local debugging. A production application should generally export telemetry to an appropriate observability backend.

Configure OpenTelemetry Tracing

Open the Program.cs file and configure tracing before building the application.

The following example demonstrates incoming request instrumentation, outgoing HTTP instrumentation, service identification, and console export.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "order-api",
            serviceVersion: "1.0.0"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter());

builder.Services.AddHttpClient();

var app = builder.Build();

app.MapGet("/", () =>
{
    return Results.Ok(new
    {
        Service = "Order API",
        Status = "Running"
    });
});

app.Run();

When the application receives an HTTP request, ASP.NET Core instrumentation creates a server-side span. When the application makes an outgoing request through an instrumented HttpClient, the HTTP instrumentation creates a corresponding client-side span.

The telemetry includes information such as request methods, routes, status codes, and operation durations, subject to the instrumentation configuration and applicable conventions.

The resource identifies the service that generated the telemetry. Setting a consistent service name is important because an observability backend needs to distinguish the order service from the payment service.

Run the application:

dotnet run

Send a request to the local endpoint using a browser or an HTTP client. The console should display trace information for the request.

At this stage, the application can trace its own HTTP activity. The next step is to connect outgoing requests to downstream services.

Trace Requests Across Multiple Services

To understand distributed tracing, consider two services:

Order API receives customer requests.
Inventory API checks whether a product is available.

The Order API calls Inventory API through HttpClient. OpenTelemetry instrumentation can create a client span for this call, while the Inventory API creates a server span when it receives the request.

When trace context is propagated correctly, the downstream span can be associated with the original trace.

Create the Inventory API

Create a second Web API project:

dotnet new webapi -n InventoryApi
cd InventoryApi

Add the necessary OpenTelemetry packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Exporter.Console

Replace the relevant content in Program.cs with:

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services
    .AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService(
            serviceName: "inventory-api",
            serviceVersion: "1.0.0"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddConsoleExporter());

var app = builder.Build();

app.MapGet("/inventory/{productId}", (string productId) =>
{
    return Results.Ok(new
    {
        ProductId = productId,
        Available = true
    });
});

app.Run();

The Inventory API now creates server-side spans for incoming HTTP requests.

For this demonstration, the endpoint returns a fixed inventory result. A production service would normally retrieve availability from a database or another inventory system.

Call Inventory API From Order API

Return to the Order API project and register an HTTP client.

builder.Services.AddHttpClient("InventoryClient", client =>
{
    client.BaseAddress = new Uri("http://localhost:5101");
});

Use the actual address and port configured for the Inventory API in your environment.

Add the following endpoint to the Order API:

app.MapGet(
    "/orders/check/{productId}",
    async (
        string productId,
        IHttpClientFactory clientFactory,
        CancellationToken cancellationToken) =>
    {
        var client = clientFactory.CreateClient("InventoryClient");

        using var response = await client.GetAsync(
            $"/inventory/{Uri.EscapeDataString(productId)}",
            cancellationToken);

        response.EnsureSuccessStatusCode();

        var inventory = await response.Content.ReadFromJsonAsync<object>(
            cancellationToken: cancellationToken);

        return Results.Ok(new
        {
            ProductId = productId,
            Inventory = inventory
        });
    });

The example uses IHttpClientFactory to create a configured client. The HTTP instrumentation observes the outgoing request, while the Inventory API instrumentation observes the incoming request.

OpenTelemetry's standard HTTP instrumentation normally propagates trace context through HTTP headers. Both services must be configured to collect the relevant spans, and their telemetry must be exported to a backend that can assemble and display the trace.

For local testing, run both services on their configured ports and request the order endpoint.

You should be able to identify the incoming Order API request and the outgoing Inventory API call in the collected telemetry.

Add Custom Spans for Business Operations

Automatic instrumentation covers common framework operations, but it does not explain every business operation.

For example, an order request might involve calculating a discount, validating inventory rules, or reserving stock. Custom spans can help developers measure these operations separately.

Use ActivitySource to create application-specific spans.

using System.Diagnostics;

public static class OrderTracing
{
    public static readonly ActivitySource ActivitySource =
        new("OrderApi.Business");
}

.WithTracing(tracing => tracing
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSource("OrderApi.Business")
    .AddConsoleExporter())

Now create a custom span around a business operation:

using var activity = OrderTracing.ActivitySource.StartActivity(
    "ValidateOrder");

activity?.SetTag("order.validation", "started");

// Perform order validation here.

activity?.SetTag("order.validation", "completed");

The span can appear as a child of the currently active request when created within the appropriate tracing context.

Custom spans should represent meaningful operations. Creating spans for every trivial method call can generate unnecessary telemetry and make traces harder to understand.

Avoid placing sensitive information such as passwords, access tokens, payment details, or personal customer data in span attributes.

Export Traces to an Observability Backend

Console output is useful during development, but it is not a practical long-term storage solution for distributed traces.

OpenTelemetry supports exporters that send telemetry to compatible monitoring systems. A common option is the OpenTelemetry Protocol, usually called OTLP.

To use an OTLP exporter, install the package:

dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

Replace the console exporter with OTLP configuration:

.WithTracing(tracing => tracing
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSource("OrderApi.Business")
    .AddOtlpExporter())

Configure the OTLP endpoint using the environment variables or application configuration expected by your deployment.

For example, an application might send telemetry to a collector running inside the same development environment. The collector can then forward traces to a compatible backend.

The endpoint must be reachable, and the exporter configuration must match the collector's protocol and security requirements.

For production systems, also configure appropriate authentication, network restrictions, retention policies, and sampling.

Sampling and Production Performance

Tracing every request can create substantial telemetry volume in high-traffic applications. Sampling determines which traces are recorded and exported.

For example, a parent-based sampling strategy can respect the sampling decision made by an upstream service. This helps maintain consistent trace collection across service boundaries.

A simplified configuration might look like this:

using OpenTelemetry.Trace;

.WithTracing(tracing => tracing
    .SetSampler(new ParentBasedSampler(
        new TraceIdRatioBasedSampler(0.10)))
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddOtlpExporter())

This configuration uses a 10% sampling ratio for traces that do not already have a parent sampling decision. It is an example, not a universal production setting.

Choose a sampling strategy based on traffic volume, diagnostic requirements, storage capacity, and the cost of losing visibility into rare failures.

Also monitor telemetry export failures and avoid adding expensive processing to request paths.

Common Mistakes to Avoid

Different Service Names

If every service reports the same service name, it becomes difficult to identify which component generated a span. Assign a stable, distinct name to each service.

Missing Trace Context

If downstream requests do not carry trace context, the backend may display separate traces instead of one connected request journey. Check instrumentation, HTTP propagation, and any custom middleware that modifies request headers.

Excessive Custom Attributes

High-cardinality attributes, such as unique customer IDs or arbitrary request values, can increase storage costs and make searches less efficient. Add attributes that help diagnose a real problem and follow your organization's privacy requirements.

Assuming Tracing Replaces Logging

Traces show the relationship and duration of operations, but they do not replace detailed application logs or metrics. Use all three signals together when investigating production problems.

Best Practices

Use consistent service names and version information.
Instrument incoming HTTP requests and outgoing dependencies.
Add custom spans for important business operations.
Propagate trace context across service boundaries.
Export telemetry to a suitable backend instead of relying on console output.
Apply sampling based on workload and diagnostic needs.
Avoid collecting secrets or sensitive personal information.
Test tracing in both successful and failing request paths.
Monitor the health of the telemetry pipeline itself.

Conclusion

OpenTelemetry makes distributed tracing practical for ASP.NET Core applications by connecting related operations across service boundaries.

With automatic HTTP instrumentation, developers can follow requests through multiple services. With custom spans, they can inspect business operations that framework instrumentation cannot describe on its own.

The most useful implementation combines consistent service identification, correct trace-context propagation, a suitable exporter, and a deliberate sampling strategy. These foundations help development teams diagnose latency, investigate failures, and understand how their microservices behave under real workloads.