Using OpenTelemetry to Trace Requests Across Serverless Functions

Serverless applications often consist of multiple independent functions that communicate through HTTP requests, queues, event streams, and managed services.

This architecture allows developers to build applications without managing traditional application servers. However, it also creates an observability challenge.

A single user request may trigger several serverless functions before the application returns a response. If one function fails or takes longer than expected, identifying the source of the problem can be difficult.

For example, an online order application might follow this workflow:

An API request reaches an order-processing function.
The function validates the order.
A payment function processes the transaction.
An inventory function updates stock.
A notification function sends a confirmation.

If the customer does not receive an order confirmation, checking only the first function's logs may not reveal the problem.

OpenTelemetry provides a standard approach to collecting distributed traces, metrics, and logs. Developers can use distributed tracing to follow a request across serverless functions and identify where delays or failures occur.

Why Distributed Tracing Matters in Serverless Applications

In a traditional monolithic application, developers can often follow a request through a relatively small number of application components.

Serverless applications distribute this work across independently executed functions.

Each function may have its own:

Execution environment.
Runtime configuration.
Logs.
Timeout settings.
Permissions.
Monitoring configuration.

A request may also cross different services and cloud infrastructure boundaries.

Without distributed tracing, developers often need to search multiple log groups and manually match timestamps, request IDs, and execution records.

Distributed tracing connects related operations into a single trace.

A trace represents the complete journey of a request, while individual operations within that trace are represented by spans.

For example:

Order API Request
       |
       v
Order Function
       |
       +------> Payment Function
       |               |
       |               v
       |          Payment API
       |
       +------> Inventory Function
       |               |
       |               v
       |          Database Update
       |
       +------> Notification Function
                       |
                       v
                  Email Service

Each operation can create a span. The related spans share trace context so that an observability platform can reconstruct the complete execution path.

Understand OpenTelemetry Traces, Spans, and Context

OpenTelemetry is an open-source observability framework that provides APIs, SDKs, and instrumentation for collecting telemetry data.

Three concepts are particularly important for distributed tracing.

Trace

A trace represents an end-to-end operation.

For example, processing one customer order can generate a trace containing several function executions and external service calls.

Each trace has a unique trace ID.

Span

A span represents an individual operation within a trace.

Examples include:

Receiving an HTTP request.
Executing a serverless function.
Calling a payment API.
Running a database query.
Publishing a message to a queue.

Spans contain information such as operation names, start times, durations, status codes, and selected attributes.

Trace context

Trace context connects spans across service boundaries.

When one function calls another service, it propagates trace information so the receiving service can continue the existing trace rather than creating an unrelated one.

The W3C Trace Context standard commonly uses headers such as:

traceparent: 00-<trace-id>-<parent-span-id>-01

The receiving service extracts this context and creates a child span.

For asynchronous messaging, the trace context should be carried in message metadata or headers supported by the messaging system.

Understand the Serverless Tracing Architecture

A typical OpenTelemetry architecture contains three main parts:

Instrumented application code.
An OpenTelemetry Collector or compatible telemetry pipeline.
A backend that stores and displays traces.

A simplified architecture looks like this:

API Gateway
     |
     v
Serverless Function A
     |
     v
Serverless Function B
     |
     v
Serverless Function C
     |
     v
OpenTelemetry Exporter
     |
     v
OpenTelemetry Collector
     |
     v
Tracing Backend

The exact deployment depends on the cloud provider.

For example, a collector may run as a sidecar, an extension, a gateway service, or a separately managed component. Some serverless platforms also provide managed integrations for exporting telemetry.

Avoid assuming that every serverless environment supports the same collector deployment model.

Implement OpenTelemetry Tracing in a .NET Application

Consider an ASP.NET Core application that receives an order request and calls another service.

The following example demonstrates basic OpenTelemetry configuration.

First, install the required packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.OpenTelemetryProtocol

The package versions should be aligned with the OpenTelemetry version supported by the application.

Next, configure tracing in Program.cs:

using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource =>
        resource.AddService(
            serviceName: "order-service"))
    .WithTracing(tracing =>
    {
        tracing
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddOtlpExporter();
    });

var app = builder.Build();

app.MapGet("/orders/{id}", async (
    string id,
    HttpClient httpClient) =>
{
    var paymentResult = await httpClient.GetAsync(
        $"https://payment-service.example/payments/{id}");

    if (!paymentResult.IsSuccessStatusCode)
    {
        return Results.Problem(
            "Payment service request failed.");
    }

    return Results.Ok(new
    {
        OrderId = id,
        Status = "Processed"
    });
});

app.Run();

This configuration enables instrumentation for incoming ASP.NET Core requests and outgoing HTTP client calls.

The OTLP exporter sends telemetry using the OpenTelemetry Protocol. The exporter must be configured with a reachable collector or compatible endpoint through the application's environment or configuration.

The example uses an illustrative payment endpoint. A real application should use its configured service address and appropriate authentication.

Create Custom Spans for Important Operations

Automatic instrumentation captures many common operations, but it does not always describe business-specific activities.

For example, an application may need to measure the time required to validate an order or calculate shipping charges.

Developers can create custom spans using the OpenTelemetry tracing API.

using System.Diagnostics;

public static class OrderTracing
{
    public static readonly ActivitySource ActivitySource =
        new("OrderService");

    public static void ValidateOrder(string orderId)
    {
        using var activity =
            ActivitySource.StartActivity("ValidateOrder");

        activity?.SetTag("order.id", orderId);

        // Perform order validation.
    }
}

tracing.AddSource("OrderService");

Now the validation operation appears as a span within the application's trace.

Use custom spans for important business operations rather than creating spans for every small method call. Excessive instrumentation can increase telemetry volume without providing useful diagnostic information.

Propagate Trace Context Between Serverless Functions

Distributed tracing works only when trace context moves between services.

For synchronous HTTP calls, supported instrumentation libraries can automatically inject and extract trace headers.

For example:

Function A
    |
    | HTTP request with traceparent
    |
    v
Function B
    |
    | Extract trace context
    |
    v
Child Span

When using standard HTTP clients and compatible OpenTelemetry instrumentation, this propagation is generally handled automatically.

However, asynchronous workflows require additional attention.

Consider an application that publishes an order event to a message queue. The message may be processed several seconds later by another serverless function.

The producer should inject the current trace context into the message metadata. The consumer should extract it before processing the message.

The exact implementation depends on the messaging system and its instrumentation support.

Do not assume that trace context will automatically survive every queue, event bus, scheduled job, or custom integration.

Handle Asynchronous Serverless Workflows

Serverless systems frequently use event-driven processing.

For example:

Order Function
      |
      v
Message Queue
      |
      v
Payment Function
      |
      v
Inventory Function
      |
      v
Notification Function

Each function may execute independently.

The tracing design should account for the relationship between the producer and consumer operations.

In some workflows, a message consumer can continue the original trace using a parent-child relationship. In other workflows, especially when processing batches or combining multiple messages, span links may represent the relationship more accurately.

The correct approach depends on the messaging pattern.

Also consider retries and dead-letter queues. A message may be processed more than once, so each processing attempt should be identifiable without losing its relationship to the original event.

Configure Sampling to Control Telemetry Volume

Distributed tracing can generate a large amount of data.

A high-traffic serverless application may produce thousands of traces every minute. Exporting every span can increase network usage, storage requirements, and observability costs.

Sampling determines which traces are collected and exported.

Common approaches include:

Always-on sampling for low-volume applications.
Probability-based sampling for high-volume systems.
Parent-based sampling to maintain consistent decisions across services.
Tail-based sampling to retain traces based on their completed characteristics, such as errors or high latency.

For example, a production application may sample a portion of successful requests while retaining a higher proportion of failed or unusually slow traces.

Tail-based sampling generally requires a collector or backend capable of making decisions after receiving the relevant spans. It should not be confused with ordinary head-based sampling performed when a trace begins.

Sampling decisions should be consistent across the distributed workflow. Otherwise, a trace may contain spans from only some of its services.

Monitor Trace Quality and Performance

Collecting traces is not enough. Developers should monitor whether the tracing system provides useful and reliable information.

Important measurements include:

Metric	Purpose
Request duration	Measures end-to-end response time
Function execution duration	Identifies slow serverless functions
Error rate	Tracks failed requests and operations
Cold-start duration	Helps identify initialization delays
Queue processing delay	Measures asynchronous workflow delays
Trace completeness	Identifies missing spans
Export failures	Detects telemetry delivery problems
Telemetry volume	Helps control storage and processing costs

Be careful when recording span attributes.

Avoid putting passwords, access tokens, full request bodies, customer records, or confidential prompts into trace metadata.

Trace data is operational data and must follow the same security, access, and retention policies as application logs.

Common Mistakes to Avoid

Instrumenting only the first function

Tracing only the API entry point does not provide visibility into downstream functions. Every relevant service boundary should participate in context propagation.

Losing trace context in message queues

If message metadata does not carry trace context, asynchronous operations may appear as unrelated traces.

Creating too many custom spans

Instrument meaningful operations. Excessive spans increase telemetry volume and make traces harder to understand.

Ignoring cold starts

A function may appear slow because of initialization rather than business logic. Use cloud-specific execution metrics alongside application traces.

Exporting sensitive information

Trace attributes can expose personal information, credentials, and confidential data. Use an allowlist of approved attributes.

Assuming traces are always complete

Sampling, exporter failures, unsupported integrations, and asynchronous processing can produce incomplete traces. Monitor telemetry pipeline health.

Best Practices for Production Serverless Tracing

A reliable implementation should follow a consistent approach:

Define service names and resource attributes consistently.
Instrument HTTP requests and outbound calls.
Create custom spans for important business operations.
Propagate trace context across synchronous and asynchronous boundaries.
Use compatible instrumentation for queues and event-driven services.
Configure sampling based on traffic and diagnostic requirements.
Monitor exporter failures and missing telemetry.
Protect trace attributes from sensitive information.
Correlate traces with application logs using trace IDs.
Review telemetry costs and retention regularly.

Also test distributed tracing during deployment and failure scenarios. A successful local trace does not guarantee that context propagation works across production gateways, queues, and serverless execution environments.

Summary

Serverless applications distribute request processing across independent functions and managed services. Without distributed tracing, identifying the source of failures and performance problems can require manually investigating multiple logs.

OpenTelemetry provides a consistent framework for collecting traces across these components.

Developers can use automatic instrumentation for common HTTP operations, custom spans for important business activities, and trace context propagation to connect synchronous and asynchronous workflows.

Sampling, telemetry monitoring, and sensitive-data controls are equally important for production deployments.

The objective is to make a complete serverless request traceable from its entry point to its final operation, giving development and operations teams the information needed to investigate failures and improve application performance.