Implementing AI Request Tracing Across Microservices with OpenTelemetry

Modern AI-powered applications rarely consist of a single service. A typical enterprise solution may include an API Gateway, authentication service, retrieval service, vector database, AI orchestration layer, Large Language Model (LLM), caching layer, and multiple downstream business services. When an AI request fails or experiences high latency, identifying the root cause can be challenging without end-to-end observability.

Traditional logging provides isolated events, but it doesn't show how a request travels across distributed systems. This is where OpenTelemetry becomes invaluable.

OpenTelemetry enables distributed tracing by assigning a unique trace to each request, allowing developers to follow an AI request from the client to every service it touches. This article explores how to implement AI request tracing in ASP.NET Core microservices using OpenTelemetry, along with practical production considerations.

Why AI Workloads Need Distributed Tracing

Unlike standard REST APIs, AI requests often involve multiple components before a response is generated.

For example:

Client
   │
API Gateway
   │
Authentication Service
   │
Prompt Builder
   │
Vector Database
   │
LLM Provider
   │
Business Service
   │
Response

A delay in any of these components can increase the overall response time. Distributed tracing helps identify where time is spent and where failures occur.

What Is OpenTelemetry?

OpenTelemetry is an open standard for collecting:

Traces
Metrics
Logs

It provides vendor-neutral instrumentation, allowing telemetry to be exported to different observability platforms.

The three primary telemetry signals are:

Signal	Purpose
Traces	Track requests across services
Metrics	Measure application performance
Logs	Capture application events

Together, these signals provide a comprehensive view of application behavior.

Understanding Trace Hierarchy

Each distributed request consists of:

One Trace
Multiple Spans

Trace
│
├── API Gateway
│
├── Authentication
│
├── Prompt Builder
│
├── Vector Search
│
├── AI Provider
│
└── Response Formatter

Each span represents an operation performed during the request lifecycle.

Adding OpenTelemetry to ASP.NET Core

builder.Services.AddOpenTelemetry()
    .WithTracing(builder =>
    {
        builder.AddAspNetCoreInstrumentation();
        builder.AddHttpClientInstrumentation();
    });

This automatically captures incoming ASP.NET Core requests and outgoing HTTP calls.

Creating Custom Activities

AI-specific operations often deserve their own spans.

private static readonly ActivitySource ActivitySource =
    new("Enterprise.AI");

using var activity = ActivitySource.StartActivity(
    "VectorSearch");

Custom activities make traces easier to understand by highlighting important business operations.

Tracing AI Provider Calls

Suppose your application sends prompts to an AI provider.

using var activity = ActivitySource.StartActivity(
    "LLMCompletion");

activity?.SetTag("ai.provider", "AzureOpenAI");
activity?.SetTag("ai.operation", "ChatCompletion");

Adding descriptive tags makes traces more useful during diagnostics.

Avoid storing sensitive prompts or personally identifiable information as trace attributes unless organizational policies explicitly allow it.

Instrumenting Vector Search

Vector retrieval is often a significant part of AI latency.

using var activity =
    ActivitySource.StartActivity("VectorSearch");

await vectorStore.SearchAsync(query);

Tracing this step separately helps distinguish retrieval delays from model inference delays.

Propagating Trace Context

Distributed tracing depends on passing trace context between services.

Gateway
    │
Trace Context
    │
Search Service
    │
Trace Context
    │
AI Service

When services propagate the trace context correctly, observability platforms can reconstruct the complete request path.

Useful AI Trace Attributes

Consider enriching spans with metadata that supports diagnostics without exposing sensitive content.

Attribute	Example
ai.provider	Azure OpenAI
ai.operation	Chat Completion
ai.model	Deployment or model identifier
ai.request.id	Correlation identifier
retrieval.source	Vector Database
response.status	Success or Failure

Choose attributes that are meaningful for your organization while respecting privacy and compliance requirements.

Measuring Latency

Distributed tracing helps identify where latency originates.

Typical breakdown:

Authentication        18 ms
Prompt Building       10 ms
Vector Search         75 ms
LLM Inference       1250 ms
Response Formatting   12 ms

These values are illustrative. Actual timings depend on infrastructure, workload, and provider performance.

Error Tracking

Capture failures within spans.

try
{
    await aiClient.GenerateAsync(prompt);
}
catch (Exception ex)
{
    activity?.SetStatus(ActivityStatusCode.Error);
    activity?.RecordException(ex);

    throw;
}

Recording exceptions directly on spans makes troubleshooting significantly easier.

Monitoring AI Pipelines

Useful operational metrics include:

End-to-end request latency
AI provider response time
Vector search duration
Request success rate
Retry count
Timeout frequency
Dependency failures

Monitoring these metrics alongside traces provides better visibility into system health.

Integrating Multiple Services

A production AI workflow may involve several services.

Client
   │
API Gateway
   │
Chat Service
   │
Retrieval Service
   │
Embedding Service
   │
Vector Database
   │
AI Provider

Each service contributes spans to the same distributed trace, creating a unified execution timeline.

Comparison of Observability Approaches

Approach	Advantages	Limitations
Logging Only	Easy to implement	Difficult to correlate across services
Metrics Only	Good for trend analysis	Limited request-level detail
Distributed Tracing	Complete request visibility	Requires instrumentation
Full Observability	Combines traces, metrics, and logs	Higher implementation effort

Most enterprise systems benefit from combining all three telemetry signals.

Common Mistakes

Mistake	Better Approach
Tracing only API requests	Instrument business operations as well
Logging entire prompts	Log metadata instead of sensitive content
Missing trace propagation	Pass context across every service boundary
Creating overly granular spans	Trace meaningful operations
Ignoring dependency instrumentation	Instrument HTTP clients, databases, and messaging where appropriate

Troubleshooting

Missing Spans

Verify:

OpenTelemetry registration
ActivitySource configuration
Instrumentation packages
Trace context propagation

Broken Trace Chains

Check that downstream services receive and forward trace context consistently.

High Trace Volume

Consider:

Sampling strategies
Filtering low-value traces
Reducing unnecessary custom spans

Balance observability needs with storage and processing costs.

Best Practices

Instrument both infrastructure and business operations.
Create meaningful span names.
Use consistent attribute naming across services.
Avoid storing confidential information in telemetry.
Monitor latency trends over time.
Review trace sampling policies as traffic grows.
Validate trace propagation during integration testing.

Conclusion

Distributed tracing is essential for understanding the behavior of AI-powered microservices. By using OpenTelemetry in ASP.NET Core, organizations can follow AI requests across gateways, retrieval services, vector databases, and language models, making it easier to diagnose latency, failures, and dependency issues.

Rather than relying solely on logs or metrics, combining distributed traces with other observability signals provides a clearer picture of how AI workloads perform in production. As AI systems become more distributed and complex, comprehensive tracing becomes an important part of building reliable, maintainable, and observable enterprise applications.

Frequently Asked Questions

Why is distributed tracing important for AI applications?

AI requests often involve multiple services and external dependencies. Distributed tracing helps identify where time is spent and where failures occur throughout the request lifecycle.

Does OpenTelemetry replace logging?

No. OpenTelemetry complements logging by providing request flow and timing information. Logs, metrics, and traces work together to support effective observability.

Should I trace every AI request?

Not necessarily. High-traffic systems often use sampling strategies to balance observability with storage and processing costs while still capturing representative traces.

Is it safe to include prompts in trace data?

Generally, avoid recording full prompts or sensitive user data in traces. If diagnostic information is required, capture non-sensitive metadata and follow your organization's security and compliance policies.