Tracing AI Agent Tool Calls with OpenTelemetry

AI agents rarely complete a task with a single operation. An agent may call a model, search a database, invoke an API, run a function, and then call the model again before returning the final answer.

When something becomes slow or fails, normal application logs may not provide enough information to understand what happened.

OpenTelemetry can help by tracing each step of an agent workflow.

Instead of seeing only:

Request completed in 8 seconds

you can see:

Agent Run
 |
 +-- Model Call       1.2s
 |
 +-- Search Tool      0.8s
 |
 +-- Database Tool    2.1s
 |
 +-- Model Call       1.5s
 |
 +-- Final Response   0.4s

This makes it easier to find slow tools, failed calls, retries, and unnecessary agent steps.

Why AI Agents Need Tracing

Traditional application tracing usually follows a request through services such as:

API
 |
 +-- Service
 |
 +-- Database

An AI agent adds another layer:

API
 |
 v
AI Agent
 |
 +-- LLM
 |
 +-- Tool
 |
 +-- Database
 |
 +-- LLM
 |
 +-- External API

The agent decides which operation to perform next.

That dynamic behavior makes tracing particularly useful because the execution path can change from one request to another.

What Should You Trace?

A useful agent trace should capture the major operations without recording sensitive content unnecessarily.

For example:

The goal is to understand the workflow rather than capture every piece of user data.

Understanding Traces and Spans

OpenTelemetry represents distributed work using traces and spans.

A trace represents the complete operation.

A span represents one operation within that trace.

For an AI agent:

Trace: Agent Request
|
+-- Agent Execution
    |
    +-- Model Call
    |
    +-- Search Tool
    |
    +-- Database Query
    |
    +-- Model Call

This hierarchy allows developers to see where time is being spent.

Creating an Agent Span in C#

For a .NET application, ActivitySource can be used to create custom spans.

using System.Diagnostics;

public static class AgentTracing
{
    public static readonly ActivitySource Source =
        new("MyCompany.AIAgent");
}

An agent operation can then be traced:

using var activity =
    AgentTracing.Source.StartActivity("agent.run");

activity?.SetTag("agent.name", "SupportAgent");

var result = await RunAgentAsync();

activity?.SetTag("agent.status", "success");

The span represents the overall agent execution.

Trace Individual Tool Calls

The most useful information often comes from tracing each tool separately.

public async Task<string> SearchAsync(string query)
{
    using var activity =
        AgentTracing.Source.StartActivity("agent.tool.search");

    activity?.SetTag("tool.name", "search");

    try
    {
        var result = await ExecuteSearchAsync(query);

        activity?.SetTag("tool.status", "success");

        return result;
    }
    catch (Exception ex)
    {
        activity?.SetTag("tool.status", "error");
        activity?.SetStatus(ActivityStatusCode.Error);

        throw;
    }
}

Now a trace can show exactly how long the search operation took and whether it failed.

Trace Model Calls Separately

Model calls should also have their own spans.

public async Task<string> CallModelAsync(string prompt)
{
    using var activity =
        AgentTracing.Source.StartActivity("agent.model");

    activity?.SetTag("model.name", "configured-model");

    var response = await SendModelRequestAsync(prompt);

    activity?.SetTag("model.status", "success");

    return response;
}

Useful metadata can include:

Model name
Input token count
Output token count
Duration
Status
Retry count

Avoid putting complete prompts or responses into traces unless there is a specific and approved reason to do so.

Connect the Spans

The real value appears when the spans form one trace.

For example:

Agent Request
|
+-- Model Call
|
+-- Search Tool
|    |
|    +-- HTTP Request
|
+-- Database Tool
|
+-- Model Call
|
+-- Final Response

Now you can identify the slow part of the workflow.

Suppose the complete request takes 7 seconds:

Model Call       1.1s
Search           0.7s
Database         4.5s
Model Call       0.7s

The database operation is clearly worth investigating.

Without tracing, the application may simply report that the agent took seven seconds.

Add Useful Attributes

Good attributes make traces easier to filter.

For example:

activity?.SetTag("agent.name", "SupportAgent");
activity?.SetTag("tool.name", "customer-search");
activity?.SetTag("tool.status", "success");
activity?.SetTag("agent.attempt", 1);

You can also record numerical values:

activity?.SetTag("gen_ai.usage.input_tokens", inputTokens);
activity?.SetTag("gen_ai.usage.output_tokens", outputTokens);

Use a consistent naming strategy across the application so dashboards and queries remain useful.

Trace Errors and Retries

Suppose a tool fails twice before succeeding:

Agent Run
 |
 +-- Search Attempt 1 - Failed
 |
 +-- Search Attempt 2 - Failed
 |
 +-- Search Attempt 3 - Success

This is much more informative than a single Search Failed log.

Record the attempt number:

activity?.SetTag("retry.attempt", attempt);

For failures, mark the span accordingly:

activity?.SetStatus(ActivityStatusCode.Error);
activity?.SetTag("error.type", ex.GetType().Name);

This makes repeated failures easier to identify.

Avoid Sensitive Trace Data

AI applications can process sensitive information.

Do not automatically put the following into telemetry:

Passwords
Access tokens
API keys
Personal information
Private documents
Complete user prompts
Complete model responses

Instead, capture metadata.

For example:

tool.name = customer-search
tool.status = success
duration_ms = 245
result_count = 12

This provides useful diagnostic information without unnecessarily copying application data into observability systems.

Measure the Right Metrics

Tracing becomes more useful when combined with metrics.

For AI agents, consider tracking:

Agent executions
Successful executions
Failed executions
Model calls
Tool calls
Tool failures
Retries
Execution duration
Token usage

For latency, look beyond the average.

Track:

P50
P95
P99

For example:

The numbers above are examples only. Production values should come from your application's telemetry.

Logs vs Traces

Logs and traces serve different purposes.

You usually want both.

For example:

Trace
 |
 +-- Tool Call
      |
      +-- Log: Database timeout

The trace tells you where the failure happened.

The log can provide additional diagnostic details.

Common Mistakes

Creating One Span for the Entire Agent

A single span does not show which tool or model call caused the delay.

Recording Too Much Data

Full prompts and responses can create privacy, security, and storage problems.

Ignoring Retries

Repeated tool calls can significantly affect latency and cost.

Using Inconsistent Names

Different names for the same tool make telemetry harder to query.

Tracking Only Errors

Successful traces are also valuable because they establish normal behavior.

Ignoring Sampling

High-volume agents can generate large amounts of telemetry. Sampling may be necessary depending on the application's scale.

Best Practices

1. Create a trace for each meaningful agent execution.

2. Create child spans for model and tool calls.

3. Record duration and status consistently.

4. Track retries and failed operations.

5. Use stable attribute names.

6. Avoid sensitive prompt and response data.

7. Combine traces with logs and metrics.

8. Monitor P95 and P99 latency.

9. Use sampling where telemetry volume requires it.

10. Keep tracing overhead small enough for production use.

Advantages

• Makes complex agent workflows easier to understand.

• Helps identify slow tools and model calls.

• Makes retry behavior visible.

• Connects errors with the operation that caused them.

• Supports production performance troubleshooting.

• Provides useful information for cost and latency optimization.

Limitations

• Detailed traces can increase telemetry volume.

• Poorly designed attributes can make analysis difficult.

• Sensitive data requires careful handling.

• High-cardinality attributes can create observability problems.

• Tracing does not fix slow operations by itself.

A Practical Agent Trace

A useful production trace might look like:

Agent Run - 6.2s
|
+-- Model Call - 1.0s
|
+-- Search Tool - 0.6s
|
+-- Database Tool - 3.4s
|    |
|    +-- Database Query - 3.2s
|
+-- Model Call - 0.9s
|
+-- Final Response - 0.3s

The trace immediately shows where most of the time was spent.

The next investigation can then focus on the database operation instead of the entire agent.

Conclusion

AI agents are dynamic workflows, and traditional application logs do not always provide enough information to understand their behavior.

OpenTelemetry tracing provides a structured way to follow an agent from the initial request through model calls, tool executions, database operations, retries, and the final response.

The most useful approach is to trace meaningful operations, record consistent metadata, avoid sensitive content, and combine traces with logs and metrics.

When an agent becomes slow or unreliable, a well-designed trace can turn a complicated workflow into a clear sequence of operations that developers can investigate.