Modern AI-powered applications rarely consist of a single service. A typical enterprise solution may include an API Gateway, authentication service, retrieval service, vector database, AI orchestration layer, Large Language Model (LLM), caching layer, and multiple downstream business services. When an AI request fails or experiences high latency, identifying the root cause can be challenging without end-to-end observability.
Traditional logging provides isolated events, but it doesn't show how a request travels across distributed systems. This is where OpenTelemetry becomes invaluable.
OpenTelemetry enables distributed tracing by assigning a unique trace to each request, allowing developers to follow an AI request from the client to every service it touches. This article explores how to implement AI request tracing in ASP.NET Core microservices using OpenTelemetry, along with practical production considerations.
Why AI Workloads Need Distributed Tracing
Unlike standard REST APIs, AI requests often involve multiple components before a response is generated.
For example:
Client
│
API Gateway
│
Authentication Service
│
Prompt Builder
│
Vector Database
│
LLM Provider
│
Business Service
│
Response
A delay in any of these components can increase the overall response time. Distributed tracing helps identify where time is spent and where failures occur.
What Is OpenTelemetry?
OpenTelemetry is an open standard for collecting:
It provides vendor-neutral instrumentation, allowing telemetry to be exported to different observability platforms.
The three primary telemetry signals are:
| Signal | Purpose |
|---|
| Traces | Track requests across services |
| Metrics | Measure application performance |
| Logs | Capture application events |
Together, these signals provide a comprehensive view of application behavior.
Understanding Trace Hierarchy
Each distributed request consists of:
Trace
│
├── API Gateway
│
├── Authentication
│
├── Prompt Builder
│
├── Vector Search
│
├── AI Provider
│
└── Response Formatter
Each span represents an operation performed during the request lifecycle.
Adding OpenTelemetry to ASP.NET Core
Register OpenTelemetry during application startup.
builder.Services.AddOpenTelemetry()
.WithTracing(builder =>
{
builder.AddAspNetCoreInstrumentation();
builder.AddHttpClientInstrumentation();
});
This automatically captures incoming ASP.NET Core requests and outgoing HTTP calls.
Creating Custom Activities
AI-specific operations often deserve their own spans.
private static readonly ActivitySource ActivitySource =
new("Enterprise.AI");
using var activity = ActivitySource.StartActivity(
"VectorSearch");
Custom activities make traces easier to understand by highlighting important business operations.
Tracing AI Provider Calls
Suppose your application sends prompts to an AI provider.
using var activity = ActivitySource.StartActivity(
"LLMCompletion");
activity?.SetTag("ai.provider", "AzureOpenAI");
activity?.SetTag("ai.operation", "ChatCompletion");
Adding descriptive tags makes traces more useful during diagnostics.
Avoid storing sensitive prompts or personally identifiable information as trace attributes unless organizational policies explicitly allow it.
Instrumenting Vector Search
Vector retrieval is often a significant part of AI latency.
using var activity =
ActivitySource.StartActivity("VectorSearch");
await vectorStore.SearchAsync(query);
Tracing this step separately helps distinguish retrieval delays from model inference delays.
Propagating Trace Context
Distributed tracing depends on passing trace context between services.
Gateway
│
Trace Context
│
Search Service
│
Trace Context
│
AI Service
When services propagate the trace context correctly, observability platforms can reconstruct the complete request path.
Useful AI Trace Attributes
Consider enriching spans with metadata that supports diagnostics without exposing sensitive content.
| Attribute | Example |
|---|
| ai.provider | Azure OpenAI |
| ai.operation | Chat Completion |
| ai.model | Deployment or model identifier |
| ai.request.id | Correlation identifier |
| retrieval.source | Vector Database |
| response.status | Success or Failure |
Choose attributes that are meaningful for your organization while respecting privacy and compliance requirements.
Measuring Latency
Distributed tracing helps identify where latency originates.
Typical breakdown:
Authentication 18 ms
Prompt Building 10 ms
Vector Search 75 ms
LLM Inference 1250 ms
Response Formatting 12 ms
These values are illustrative. Actual timings depend on infrastructure, workload, and provider performance.
Error Tracking
Capture failures within spans.
try
{
await aiClient.GenerateAsync(prompt);
}
catch (Exception ex)
{
activity?.SetStatus(ActivityStatusCode.Error);
activity?.RecordException(ex);
throw;
}
Recording exceptions directly on spans makes troubleshooting significantly easier.
Monitoring AI Pipelines
Useful operational metrics include:
Monitoring these metrics alongside traces provides better visibility into system health.
Integrating Multiple Services
A production AI workflow may involve several services.
Client
│
API Gateway
│
Chat Service
│
Retrieval Service
│
Embedding Service
│
Vector Database
│
AI Provider
Each service contributes spans to the same distributed trace, creating a unified execution timeline.
Comparison of Observability Approaches
| Approach | Advantages | Limitations |
|---|
| Logging Only | Easy to implement | Difficult to correlate across services |
| Metrics Only | Good for trend analysis | Limited request-level detail |
| Distributed Tracing | Complete request visibility | Requires instrumentation |
| Full Observability | Combines traces, metrics, and logs | Higher implementation effort |
Most enterprise systems benefit from combining all three telemetry signals.
Common Mistakes
| Mistake | Better Approach |
|---|
| Tracing only API requests | Instrument business operations as well |
| Logging entire prompts | Log metadata instead of sensitive content |
| Missing trace propagation | Pass context across every service boundary |
| Creating overly granular spans | Trace meaningful operations |
| Ignoring dependency instrumentation | Instrument HTTP clients, databases, and messaging where appropriate |
Troubleshooting
Missing Spans
Verify:
OpenTelemetry registration
ActivitySource configuration
Instrumentation packages
Trace context propagation
Broken Trace Chains
Check that downstream services receive and forward trace context consistently.
High Trace Volume
Consider:
Balance observability needs with storage and processing costs.
Best Practices
Instrument both infrastructure and business operations.
Create meaningful span names.
Use consistent attribute naming across services.
Avoid storing confidential information in telemetry.
Monitor latency trends over time.
Review trace sampling policies as traffic grows.
Validate trace propagation during integration testing.
Conclusion
Distributed tracing is essential for understanding the behavior of AI-powered microservices. By using OpenTelemetry in ASP.NET Core, organizations can follow AI requests across gateways, retrieval services, vector databases, and language models, making it easier to diagnose latency, failures, and dependency issues.
Rather than relying solely on logs or metrics, combining distributed traces with other observability signals provides a clearer picture of how AI workloads perform in production. As AI systems become more distributed and complex, comprehensive tracing becomes an important part of building reliable, maintainable, and observable enterprise applications.
Frequently Asked Questions
Why is distributed tracing important for AI applications?
AI requests often involve multiple services and external dependencies. Distributed tracing helps identify where time is spent and where failures occur throughout the request lifecycle.
Does OpenTelemetry replace logging?
No. OpenTelemetry complements logging by providing request flow and timing information. Logs, metrics, and traces work together to support effective observability.
Should I trace every AI request?
Not necessarily. High-traffic systems often use sampling strategies to balance observability with storage and processing costs while still capturing representative traces.
Is it safe to include prompts in trace data?
Generally, avoid recording full prompts or sensitive user data in traces. If diagnostic information is required, capture non-sensitive metadata and follow your organization's security and compliance policies.