Table of Contents

  1. What is OpenTelemetry?
  2. Setting Up OpenTelemetry in ASP.NET Core
    • Metrics
    • Logging
    • Tracing
    • Exporting to Azure Monitor
  3. Setting Up Azure Application Insights, Alerts, and Monitoring
  4. Viewing Custom Metrics in Azure Monitor
  5. Query Custom Metrics with KQL
  6. Deep Dive into Telemetry Signals
    • Distributed Tracing
    • Metrics
    • Logs
  7. Advanced Telemetry Patterns
  8. Conclusion

What is OpenTelemetry?

OpenTelemetry is a vendor-neutral instrumentation framework for generating, collecting, processing, and exporting telemetry data such as traces, metrics, and logs. It provides unified APIs and SDKs to help developers monitor applications effectively.

Setting Up OpenTelemetry in ASP.NET Core

To start, you need to install the required NuGet packages:

dotnet add package OpenTelemetry.Extensions.Hosting
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.Http
dotnet add package OpenTelemetry.Instrumentation.Runtime
dotnet add package Azure.Monitor.OpenTelemetry.Exporter

Metrics

Metrics help track the behavior and performance of an application over time.

var meter = new Meter("MyCustomMeter", "1.0");
var counter = meter.CreateCounter<int>("my_custom_counter");
counter.Add(1, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));

You can also use Histogram to measure latency:

var sw=Stopwatch.StartNew();
var meter = new Meter("MyCustomMeter", "1.0");
var requestDuration = meter.CreateHistogram<double>("myApp.request_duration", unit: "ms");
  var result= Enumerable.Range(1, 5).Select(index => new WeatherForecast
  {
      Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
      TemperatureC = Random.Shared.Next(-20, 55),
      Summary = Summaries[Random.Shared.Next(Summaries.Length)]
  })
  .ToArray();
 sw.Stop();
 requestDuration.Record(sw.ElapsedMilliseconds, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));

Logging

To log-structured data:

_logger.LogInformation("Request duration: {@Duration} ms", sw.ElapsedMilliseconds);

If you use @ before the object name, it ensures structured logging, preserving the shape of the object.

Tracing

To enable distributed tracing:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
            .AddSource("opentelemetryverify")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddConsoleExporter()
            .AddAzureMonitorTraceExporter(o =>
            {
                // Set the connection string for Azure Monitor
                o.ConnectionString = connectionString;
            });
    });

Exporting to Azure Monitor

To send telemetry data to Azure Monitor:

  1. Create an Application Insights resource in Azure.
  2. Copy the connection string from the Overview blade.
  3. In appsettings.json or secrets:
"APPLICATIONINSIGHTS_CONNECTION_STRING": "InstrumentationKey=..."

Configure exporters:

builder.Services.AddOpenTelemetryMetrics(builder =>
{
    builder.AddAspNetCoreInstrumentation()
           .AddHttpClientInstrumentation()
           .AddAzureMonitorMetricExporter(options =>
           {
               options.ConnectionString = configuration["APPLICATIONINSIGHTS_CONNECTION_STRING"];
           });
});

Setting Up Azure Application Insights, Alerts, and Monitoring


1. Application Insights Setup

2. Connect Your App

Paste the connection string into your application’s configuration and use Azure Monitor exporters in the OpenTelemetry setup.

3. Custom Metrics

4. Alerts Configuration

5. Workbooks and Dashboards

6. Monitoring with Log Analytics

You can write custom queries for logs and metrics using KQL (Kusto Query Language):

customMetrics
| where name == "myapp.request_count"
| summarize count() by bin(timestamp, 1h), name
| render timechart

Viewing Custom Metrics in Azure Monitor

To view metrics:

Query Custom Metrics with KQL

In Azure Monitor Logs:

customMetrics
| where name == "myapp.request_count"
| summarize count() by bin(timestamp, 1h), name
| render timechart

Deep Dive into Telemetry Signals

OpenTelemetry provides three core types of signals for observability: Traces, Metrics, and Logs. Each serves a different purpose, but together, they provide a complete view of your application’s health and performance.

🧵 Distributed Tracing

Purpose: Traces help you understand the journey of a request as it flows through various components and services in a distributed system.

Key Concepts

Example Use Case

You’re debugging why a request is slow. A trace shows that while your API responded quickly, the delay was due to a downstream database call taking too long.

Code

builder.Services.AddOpenTelemetry()
    .WithTracing(tracerProviderBuilder =>
    {
        tracerProviderBuilder
        .AddSource("opentelemetryverify")
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddConsoleExporter()
            .AddAzureMonitorTraceExporter(o =>
            {
                o.ConnectionString = connectionString; // Set the connection string for Azure Monitor
            });
    })

Metrics

Purpose: Metrics provide quantitative information over time to measure application performance and health.

Key Metric Instruments

Example Use Case

You want to monitor request rates and latency. Metrics help you visualize traffic patterns and performance trends over time.

var sw=Stopwatch.StartNew();
var meter = new Meter("MyCustomMeter", "1.0");
var requestDuration = meter.CreateHistogram<double>("myApp.request_duration", unit: "ms");
var counter = meter.CreateCounter<int>("my_custom_counter");
counter.Add(1, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));
 
var dbConnection = meter.CreateUpDownCounter<int>("dbconnection_count");
dbConnection.Add(1);
var result= Enumerable.Range(1, 5).Select(index => new WeatherForecast
{
    Date = DateOnly.FromDateTime(DateTime.Now.AddDays(index)),
    TemperatureC = Random.Shared.Next(-20, 55),
    Summary = Summaries[Random.Shared.Next(Summaries.Length)]
})
.ToArray();
dbConnection.Add(-1);
sw.Stop();
requestDuration.Record(sw.ElapsedMilliseconds, new KeyValuePair<string, object?>("path", Request.HttpContext.Request));
_logger.LogInformation("Request duration: {Duration} ms", sw.ElapsedMilliseconds);

WithMetrics(metrics =>
{
    metrics
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddHttpClientInstrumentation()
        .AddMeter("MyCustomMeter") // Optional: For custom metrics
        .AddConsoleExporter()     // Export metrics to console
        .AddAzureMonitorMetricExporter(o =>
        {
            o.ConnectionString = connectionString; // Set the connection string for Azure Monitor
        });
});

Logs

Purpose: Logs provide contextual, human-readable messages about events or errors occurring in your app.

Structured Logging

Logging with structured data (objects, dictionaries) makes it easier to search and analyze logs in Azure Monitor or any observability platform.

Example Use Case

You log a response payload for auditing or an exception trace for troubleshooting. Structured logs help you query specific fields like status codes or response times.

Code

builder.Logging.ClearProviders();
builder.Logging.AddOpenTelemetry(x =>
{
    x.SetResourceBuilder(ResourceBuilder.CreateDefault()
        .AddService("OpenTelemetryVerify", serviceVersion: "1.0.0")); // Set the service name and version
    x.IncludeFormattedMessage = true; // Include formatted message in logs
    x.IncludeScopes = true; // Include scopes in logs
    x.ParseStateValues = true; // Parse state values in logs
    x.AddConsoleExporter(); // Export logs to console
    x.AddAzureMonitorLogExporter(o =>
    {
        o.ConnectionString = connectionString; // Set the connection string for Azure Monitor
    });
});
_logger.LogInformation("Response of the weather forecast:{@result}", data);

Bonus Tip

To view structured logs correctly in Azure Monitor, ensure your logger supports scopes and enrichment (e.g., with Serilog or OpenTelemetry exporters).

Advanced Telemetry Patterns

1. Correlation Across Services

Correlate telemetry data using TraceId and SpanId across services to track a request’s flow. This enables full end-to-end visibility in distributed systems.

2. Redaction and Enrichment

Use processors to add metadata like userId or region and to remove or mask sensitive fields before telemetry is exported.

3. Sampling Strategies

Reduce data volume by configuring sampling. Use TraceIdRatioBasedSampler(0.1) to export 10% of traces while keeping error traces always.

4. Span Events and Annotations

Add custom ActivityEvents to spans to track the lifecycle stages of an operation (e.g., “CacheMiss”, “PaymentValidated”).

5. Custom Instrumentation

Instrument your domain-specific workflows like Checkout or VerifyOTP with custom spans or metrics. This gives business-level observability.

6. Error and Exception Tracking

Auto-capture exceptions in spans and logs. Use AddExceptionInstrumentation() to bind exception telemetry with traces.

7. Real-Time Alerting

Leverage Azure Monitor alerts based on custom metrics or logs. Use KQL queries to define thresholds and trigger notifications.

8. Dashboards and SLO Monitoring

Build dashboards using Azure Workbooks to monitor SLIs (e.g., error rate < 1%) and SLOs (e.g., 95th percentile latency < 300ms).

9. Cold Start and Warm Path Differentiation

Track cold start delays in serverless or autoscaling systems by tagging spans with startup metadata.

10. Multi-Exporter Strategy

Send telemetry to multiple backends (e.g., Azure Monitor and local file) for redundancy or migration testing.

Conclusion

By integrating OpenTelemetry in ASP.NET Core and exporting to Azure Monitor, you gain deep insights into your application using logs, metrics, and traces. This setup ensures robust observability, enabling proactive monitoring and faster issue resolution.

With advanced telemetry patterns, you can enrich and extend your observability to meet the demands of modern, distributed, and dynamic systems.