How to Trace a Slow API Request with OpenTelemetry in .NET

Introduction

Have you ever received a bug report saying that an API is taking too long to respond, but you cannot figure out where the delay is happening?

The request might spend time executing application code, waiting for a database query, calling another API, or processing data before returning a response. Looking only at the total response time does not tell you which part is responsible.

This becomes even harder in a microservices application, where one request may pass through several services before reaching the final response.

OpenTelemetry helps you investigate these problems by collecting traces, metrics, and logs from your application. With distributed tracing, you can follow a request through different operations and see how much time each operation takes.

In this article, we will set up OpenTelemetry in an ASP.NET Core Web API, trace incoming requests and database operations, and learn how to use trace data to investigate slow API responses.

What Is OpenTelemetry?

OpenTelemetry is an open-source framework for generating, collecting, and exporting telemetry data from applications.

It supports three main types of telemetry:

Traces: Show the path a request takes through an application and its dependencies.
Metrics: Provide numerical measurements such as request duration, request count, and error rate.
Logs: Record events and messages that help explain what happened during execution.

For investigating a slow API request, distributed tracing is usually the best starting point.

A trace contains one or more spans. Each span represents an operation, such as handling an HTTP request, executing a database command, or calling another service.

For example, a request might look like this:

GET /api/orders/101
    |
    |-- Load order from database       120 ms
    |
    |-- Call payment service           850 ms
    |
    |-- Build response                  30 ms

These timings are illustrative, not benchmark results. In a real trace, the payment service call would be a strong candidate for investigation because it accounts for most of the displayed duration.

Step 1: Create an ASP.NET Core Web API

Create a new Web API project if you do not already have one.

dotnet new webapi -n OpenTelemetryDemo
cd OpenTelemetryDemo

Next, install the OpenTelemetry packages required for HTTP request tracing and console output.

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.Console

These packages provide the hosting integration, automatic instrumentation for incoming ASP.NET Core requests, instrumentation for outgoing HTTP calls, and a console exporter for inspecting telemetry during development.

Use compatible package versions for your target .NET framework.

Step 2: Configure OpenTelemetry Tracing

Open Program.cs and configure OpenTelemetry.

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("OrdersApi"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter());

builder.Services.AddControllers();

var app = builder.Build();

app.UseHttpsRedirection();

app.MapControllers();

app.Run();

Let's look at the important parts.

AddService("OrdersApi") identifies the application in the telemetry data. In a microservices environment, each service should have a clear and consistent name.

AddAspNetCoreInstrumentation() creates spans for incoming HTTP requests handled by ASP.NET Core.

AddHttpClientInstrumentation() captures outgoing HTTP requests made through the .NET HttpClient infrastructure.

AddConsoleExporter() writes collected spans to the console. This is useful when learning how tracing works, but a production application should generally export telemetry to a dedicated observability backend.

Once the application starts, make a few API requests. You should see trace information in the console when the corresponding spans are recorded.

Step 3: Create an API Endpoint to Trace

Now create a controller that simulates an API operation involving an outgoing HTTP request.

Create a file named OrdersController.cs.

using Microsoft.AspNetCore.Mvc;

namespace OpenTelemetryDemo.Controllers;

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private readonly HttpClient _httpClient;

    public OrdersController(HttpClient httpClient)
    {
        _httpClient = httpClient;
    }

    [HttpGet("{id:int}")]
    public async Task<IActionResult> GetOrder(
        int id,
        CancellationToken cancellationToken)
    {
        var order = new
        {
            Id = id,
            ProductName = "Laptop",
            TotalAmount = 75000
        };

        var response = await _httpClient.GetAsync(
            "https://example.com/",
            cancellationToken);

        response.EnsureSuccessStatusCode();

        return Ok(order);
    }
}

This example demonstrates an incoming request followed by an outgoing HTTP call. The external address is only a placeholder; replace it with an appropriate endpoint before testing.

builder.Services.AddHttpClient<OrdersController>();

Place this registration before builder.Build().

When you call /api/orders/101, OpenTelemetry can record the incoming request and the outgoing HTTP operation.

The example uses a simple in-memory order to keep the focus on tracing. In a real application, the database access would normally live in a service or repository.

Also, an external call should not be added to an endpoint merely to generate telemetry. Instrument the dependencies that the application genuinely uses.

Step 4: Add Custom Spans to Important Operations

Automatic instrumentation covers many common operations, but it does not explain every part of your business logic.

Suppose you want to measure how long your application spends preparing an order response. You can create a custom activity using ActivitySource.

First, create an activity source in a separate class.

using System.Diagnostics;

public static class OrderTelemetry
{
    public const string SourceName = "OpenTelemetryDemo.Orders";

    public static readonly ActivitySource ActivitySource =
        new(SourceName);
}

Next, register the source with OpenTelemetry.

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("OrdersApi"))
    .WithTracing(tracing => tracing
        .AddSource(OrderTelemetry.SourceName)
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddConsoleExporter());

If you already have an OpenTelemetry registration, update that registration instead of adding a second independent configuration unnecessarily.

Now add a custom span around the operation you want to measure.

[HttpGet("{id:int}")]
public async Task<IActionResult> GetOrder(
    int id,
    CancellationToken cancellationToken)
{
    using var activity = OrderTelemetry.ActivitySource
        .StartActivity("LoadOrderDetails");

    activity?.SetTag("order.id", id);

    var order = await LoadOrderAsync(
        id,
        cancellationToken);

    return Ok(order);
}

private static Task<object> LoadOrderAsync(
    int id,
    CancellationToken cancellationToken)
{
    cancellationToken.ThrowIfCancellationRequested();

    object order = new
    {
        Id = id,
        ProductName = "Laptop",
        TotalAmount = 75000
    };

    return Task.FromResult(order);
}

This example uses a placeholder method for loading order details. In a real application, place the activity around the actual database or business operation.

StartActivity() creates a span when an appropriate listener is active. OpenTelemetry listens to the registered source and exports the span.

The order.id tag demonstrates how to attach useful context. Avoid adding customer information, credentials, payment details, or other sensitive values to telemetry.

Step 5: Trace Database Operations

Database calls are a common source of slow API responses. However, tracing the incoming HTTP request alone does not automatically provide detailed timing for every database command.

The instrumentation you need depends on your database access technology.

For applications using Entity Framework Core, you can add the EF Core instrumentation package.

dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore

Then register the instrumentation with the tracing configuration:

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("OrdersApi"))
    .WithTracing(tracing => tracing
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddConsoleExporter());

As with the previous examples, add this to your existing OpenTelemetry configuration rather than duplicating the setup.

EF Core instrumentation can help capture database-related operations, depending on the instrumentation package and the operations being performed.

Review the package's supported behavior and configuration for your EF Core version. If you need detailed SQL command timings, database-specific instrumentation or carefully configured command logging may also be necessary.

Do not assume that every database query will automatically appear as a separate span in every configuration.

How to Find the Cause of a Slow Request

Once your application exports traces to an observability backend, investigate a slow request systematically.

1. Find the slow request

Filter traces by the endpoint, service name, response status, or duration. Select a request whose response time is noticeably higher than expected.

2. Inspect the trace timeline

Look at the parent span for the HTTP request and its child spans. Check which operations consume the most time.

3. Check database and external service calls

If a database operation takes most of the request time, investigate its SQL query, execution plan, indexes, and the amount of data retrieved.

If an outgoing HTTP request is slow, investigate the remote service, network latency, timeouts, and retry behavior.

4. Compare multiple requests

A single trace may represent an unusual event. Compare several slow requests with normal requests to identify recurring patterns.

5. Check errors and retries

A request may become slow because an operation fails and is retried several times. Look at span status, exception information, and retry-related events when those are available.

Keep in mind that span durations can overlap when operations run concurrently. Do not simply add every child span duration together and assume that the total equals the request duration.

Best Practices for Production Applications

Use a dedicated telemetry backend

Console output is useful during development, but it is not a practical long-term solution for distributed production tracing. Export telemetry to an observability platform that supports trace search, timelines, filtering, and service relationships.

Configure sampling carefully

High-traffic applications can generate a large amount of telemetry. Sampling helps control the volume of traces collected, but aggressive sampling may hide rare slow requests. Choose a sampling strategy that matches your diagnostic requirements.

Avoid sensitive data in spans

Do not record passwords, access tokens, connection strings, payment information, or unnecessary personal data. Review the tags and attributes added by both automatic and custom instrumentation.

Use consistent service names

Give each service a stable, recognizable name. Consistent resource attributes make it easier to filter traces across multiple services.

Monitor latency alongside errors

Tracing helps explain individual requests, while metrics help identify broader patterns. Track request duration, error rates, and request volume so you can identify when further trace investigation is needed.

Common Mistakes and Troubleshooting

Problem	Possible reason	What to check
No traces appear	Instrumentation or exporter is missing	Verify package installation and registration
Custom spans are missing	The activity source is not registered	Add the source name with `AddSource()`
Outgoing HTTP calls are not traced	Calls bypass supported `HttpClient` instrumentation	Review how the HTTP client is created and used
Database timings are missing	Database instrumentation is not configured or supported for the operation	Check the instrumentation package and database provider
Too much telemetry is collected	Sampling and filtering are not configured	Review sampling and export settings
Requests contain sensitive attributes	Automatic or custom instrumentation captures unwanted data	Review recorded tags and sanitize telemetry

Another common mistake is expecting OpenTelemetry to fix performance problems automatically. It does not optimize SQL queries, reduce network latency, or improve business logic by itself. It provides the evidence you need to identify and investigate those problems.

Advantages and Disadvantages of OpenTelemetry

Advantages

Helps identify which operations contribute to slow API responses.
Supports tracing across multiple services.
Integrates with ASP.NET Core and common .NET dependencies.
Allows custom spans for important business operations.
Provides a common telemetry framework that supports different backends.

Disadvantages

Requires configuration and an appropriate telemetry backend for production use.
Generates additional processing and data-export overhead.
Needs careful sampling and retention policies at high traffic volumes.
Does not automatically instrument every custom operation or database provider.

Summary

OpenTelemetry makes it easier to investigate slow API requests in ASP.NET Core applications. It provides a view of how requests move through your application and its dependencies, helping you identify operations that deserve closer investigation.

Start with ASP.NET Core and HTTP client instrumentation. Add custom spans around important business operations, and configure database instrumentation when you need to investigate query performance.

For production applications, export traces to a suitable observability backend, configure sampling carefully, and avoid collecting sensitive information.

Most importantly, use traces to find the actual bottleneck before changing the code. That approach makes performance troubleshooting more focused and helps you avoid optimizing the wrong part of the application.