Production Observability: Structured Logging, Metrics, and Distributed Tracing

When an ASP.NET Core Web API encounters an unhandled exception or performance bottleneck in a high-traffic production environment, traditional debugging methods (such as attaching a debugger or reading flat console text strings) fall short. Without deep observability, diagnosing intermittent failures across distributed services becomes a guessing game.

Production Observability relies on the "Three Pillars":

Logs: Immutable, time-stamped records of discrete system events.
Metrics: Aggregated numerical counters and gauges measuring system performance (e.g., CPU usage, request rate, error latency percentiles).
Traces: End-to-end tracking of a single request as it propagates across multiple internal services and databases.

This article walks through a complete, production-grade implementation of structured logging with Serilog and distributed tracing using OpenTelemetry in an ASP.NET Core Web API.

1. Step 1: Installing Observability NuGet Packages

Install Serilog for structured logging and OpenTelemetry packages for distributed tracing into your API project:

Shell

dotnet add package Serilog.AspNetCore
dotnet add package Serilog.Sinks.Console
dotnet add package Serilog.Sinks.File
dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Exporter.Console

2. Step 2: Configuring Structured Logging with Serilog

Flat text logs like Console.WriteLine("User " + userId + " failed login") are difficult to query in log aggregators (like Grafana Loki, Seq, or Datadog). Structured logging preserves objects and key-value properties so you can filter logs by dynamic parameters.

Configure Serilog early in Program.cs:

// WebApi/Program.cs
using Serilog;

// 1. Configure bootstrap logger for startup errors
Log.Logger = new LoggerConfiguration()
    .WriteTo.Console()
    .CreateBootstrapLogger();

try
{
    var builder = WebApplication.CreateBuilder(args);

    // 2. Replace default logging with Serilog
    builder.Host.UseSerilog((context, services, configuration) => configuration
        .ReadFrom.Configuration(context.Configuration)
        .ReadFrom.Services(services)
        .Enrich.FromLogContext()
        .Enrich.WithMachineName()
        .Enrich.WithThreadId()
        .WriteTo.Console(outputTemplate: "[{Timestamp:HH:mm:ss} {Level:u3}] {Message:lj} {Properties:j}{NewLine}{Exception}")
        .WriteTo.File("logs/api-log-.txt", rollingInterval: RollingInterval.Day, restrictedToMinimumLevel: Serilog.Events.LogEventLevel.Warning));

    builder.Services.AddControllers();
    builder.Services.AddEndpointsApiExplorer();
    builder.Services.AddSwaggerGen();

    // 3. Configure OpenTelemetry for Distributed Tracing
    builder.Services.AddOpenTelemetry()
        .WithTracing(tracerProviderBuilder =>
        {
            tracerProviderBuilder
                .AddSource("MyApp.Api")
                .SetResourceBuilder(Microsoft.OpenTelemetry.Resources.ResourceBuilder.CreateDefault().AddService("ProductAPI"))
                .AddAspNetCoreInstrumentation()
                .AddHttpClientInstrumentation()
                .AddConsoleExporter(); // Can be swapped out for OTLP exporter pointing to Grafana/Jaeger
        });

    var app = builder.Build();

    app.UseSerilogRequestLogging(); // Automatically logs HTTP requests with timing and status codes

    if (app.Environment.IsDevelopment())
    {
        app.UseSwagger();
        app.UseSwaggerUI();
    }

    app.UseHttpsRedirection();
    app.UseAuthorization();
    app.MapControllers();

    app.Run();
}
v catch (Exception ex)
{
    Log.Fatal(ex, "Application terminated unexpectedly");
}
finally
{
    Log.CloseAndFlush();
}

3. Step 3: Writing Structured Logs with Properties

When writing logs inside your services or controllers, avoid string interpolation ($""). Instead, use template parameter syntax so Serilog can capture properties as queryable metadata.

// Application/Products/Commands/CreateProductCommandHandler.cs
using Domain.Entities;
using Domain.Interfaces;
using MediatR;
using Microsoft.Extensions.Logging;

namespace Application.Products.Commands;

public class CreateProductCommandHandler : IRequestHandler<CreateProductCommand, int>
{
    private readonly IProductRepository _productRepository;
    private readonly ILogger<CreateProductCommandHandler> _logger;

    public CreateProductCommandHandler(
        IProductRepository productRepository, 
        ILogger<CreateProductCommandHandler> logger)
    {
        _productRepository = productRepository;
        _logger = logger;
    }

    public async Task<int> Handle(CreateProductCommand request, CancellationToken cancellationToken)
    {
        // Notice the structured message template: {ProductName} and {Price}
        _logger.LogInformation("Creating new product entry for {ProductName} with price {Price}", request.Name, request.Price);

        var product = new Product(request.Name, request.Price, request.Stock);
        await _productRepository.AddAsync(product, cancellationToken);

        _logger.LogInformation("Successfully created product with ID {ProductId}", product.Id);

        return product.Id;
    }
}

4. Step 4: Tracking Correlation IDs Across Distributed Calls

In a microservices or distributed environment, requests pass through multiple gateways and APIs. A Correlation ID (or Trace ID) tracks a single request across all systems.

ASP.NET Core and OpenTelemetry automatically inject and propagate standard traceparent headers into incoming and outgoing HTTP requests. You can also enrich Serilog logs with this trace context automatically:

// Optional: Serilog enricher configuration automatically pulls TraceId if OpenTelemetry is active
// No manual middleware required when using OpenTelemetry + Serilog integration!

When an error occurs, you can search your log aggregator (like Grafana Loki or Seq) for the specific TraceId or ProductId property instantly, viewing the entire chronological timeline of execution.

Conclusion

By implementing structured logging with Serilog, request telemetry logging via UseSerilogRequestLogging(), and distributed tracing with OpenTelemetry, your ASP.NET Core Web API transforms from a black box into a fully transparent, highly monitorable production service.