AI Agent Observability in .NET: Tracking Tool Calls, Tokens, Latency, and Failures

AI agents introduce a new type of application behavior. Instead of executing a fixed sequence of operations, an agent may interpret a request, select tools, retrieve information, call external services, make additional model requests, and then generate a final response.

Traditional application logging is often not enough to understand what happened during such an execution.

For production .NET applications, teams need visibility into the complete agent workflow, including model calls, tool calls, token usage, latency, errors, retries, and final outcomes.

This article demonstrates how to design an observability layer for AI agents using standard .NET logging concepts and OpenTelemetry-compatible tracing patterns.

Why AI Agent Observability Is Different

A traditional API request may look like:

HTTP Request
     ↓
Controller
     ↓
Service
     ↓
Database
     ↓
HTTP Response

An AI agent may perform a much less predictable sequence:

User Request
     ↓
Agent
     ↓
Model Call
     ↓
Tool Selection
     ↓
Search Tool
     ↓
Model Call
     ↓
Database Tool
     ↓
Model Call
     ↓
Final Response

If the final response is incorrect, simply logging:

Request failed

does not explain why.

Observability should answer questions such as:

Which model was called?
How many model calls occurred?
Which tools were invoked?
How long did each operation take?
How many tokens were consumed?
Where did the failure occur?
Was a retry performed?
What was the final outcome?

The Four Core Metrics

A useful starting point is to track four categories.

Category	Examples
Tool usage	Tool name, invocation count, result
Token usage	Input and output tokens
Latency	Model, tool, and total duration
Failures	Exceptions, status, retry count

These metrics should be correlated to a single agent execution.

Create an Agent Execution ID

Every agent request should have a correlation identifier.

var executionId = Guid.NewGuid();

logger.LogInformation(
    "Agent execution started: {ExecutionId}",
    executionId);

Use the same identifier across the agent's operations.

Conceptually:

Execution ID: 8c1...
       |
       +---- Model Call
       |
       +---- Tool Call
       |
       +---- Model Call
       |
       +---- Final Response

This makes debugging multi-step execution significantly easier.

Use Structured Logging

Avoid unstructured messages such as:

logger.LogInformation(
    $"Model took {duration} milliseconds.");

Prefer structured logging:

logger.LogInformation(
    "Model call completed in {DurationMs} ms",
    duration.TotalMilliseconds);

Structured properties can then be indexed and queried by your observability platform.

Useful fields include:

ExecutionId
Model
Operation
DurationMs
InputTokens
OutputTokens
ToolName
Success
ErrorType

Representing Model Usage

Create a simple model for usage information:

public sealed record ModelUsage(
    string Model,
    long InputTokens,
    long OutputTokens)
{
    public long TotalTokens =>
        InputTokens + OutputTokens;
}

This keeps token calculations centralized.

For example:

var usage = new ModelUsage(
    "example-model",
    4200,
    850);

logger.LogInformation(
    "Model usage: {InputTokens} input, " +
    "{OutputTokens} output, {TotalTokens} total",
    usage.InputTokens,
    usage.OutputTokens,
    usage.TotalTokens);

The actual usage fields available depend on the AI provider and SDK.

Do not fabricate token values when the provider does not expose them.

Tracking Tool Calls

Tool calls should be treated as first-class operations.

Define a record:

public sealed record ToolExecution(
    string Name,
    TimeSpan Duration,
    bool Succeeded);

A tool wrapper can capture execution information:

var stopwatch = Stopwatch.StartNew();

try
{
    var result =
        await tool.ExecuteAsync(
            cancellationToken);

    stopwatch.Stop();

    logger.LogInformation(
        "Tool {ToolName} completed in {DurationMs} ms",
        tool.Name,
        stopwatch.Elapsed.TotalMilliseconds);

    return result;
}
catch (Exception ex)
{
    stopwatch.Stop();

    logger.LogError(
        ex,
        "Tool {ToolName} failed after {DurationMs} ms",
        tool.Name,
        stopwatch.Elapsed.TotalMilliseconds);

    throw;
}

This creates a consistent observability boundary around tool execution.

Tracking Latency

Total latency alone is not enough.

Consider:

Total Agent Duration = 8 seconds

Model Calls = 5 seconds
Tool Calls  = 2 seconds
Other Work  = 1 second

The model is the largest contributor.

But another request might look like:

Total Agent Duration = 8 seconds

Model Calls = 2 seconds
Database Tool = 5 seconds
Other Work = 1 second

The optimization priorities are completely different.

Track latency independently for:

Total agent execution
Model calls
Tool calls
External HTTP requests
Database operations

Use Stopwatch for Local Measurement

For simple application-level measurement:

var stopwatch = Stopwatch.StartNew();

await ExecuteAgentAsync(
    cancellationToken);

stopwatch.Stop();

logger.LogInformation(
    "Agent completed in {DurationMs} ms",
    stopwatch.Elapsed.TotalMilliseconds);

For distributed systems, tracing is generally more useful because it provides relationships between operations across service boundaries.

OpenTelemetry Tracing

OpenTelemetry provides a standardized approach for collecting distributed traces and metrics.

Create an activity source:

using System.Diagnostics;

public static class AgentTelemetry
{
    public static readonly ActivitySource Source =
        new("CustomerAgent");
}

Create an activity around an agent execution:

using var activity =
    AgentTelemetry.Source.StartActivity(
        "agent.execute");

activity?.SetTag(
    "agent.execution_id",
    executionId.ToString());

activity?.SetTag(
    "agent.operation",
    "customer-support");

This gives observability systems a trace/span representation of the operation.

Model Calls as Child Spans

A model call can become a child operation:

using var activity =
    AgentTelemetry.Source.StartActivity(
        "model.call");

activity?.SetTag(
    "gen_ai.request.model",
    modelName);

var response =
    await modelClient.SendAsync(
        request,
        cancellationToken);

Depending on the provider and instrumentation library, additional AI-related semantic attributes may be available.

Avoid assuming that every provider exposes identical telemetry fields.

Tool Calls as Child Spans

The trace can then represent:

agent.execute
│
├── model.call
│
├── tool.search
│
├── model.call
│
├── tool.database
│
└── model.call

This is much more useful than a collection of unrelated log entries.

If an agent takes 12 seconds, the trace can reveal which operation consumed most of the time.

Record Failures as Structured Events

Do not simply log:

Agent failed.

Capture useful diagnostic information:

activity?.SetTag(
    "agent.success",
    false);

activity?.SetTag(
    "error.type",
    ex.GetType().Name);

logger.LogError(
    ex,
    "Agent execution failed. ExecutionId: {ExecutionId}",
    executionId);

Avoid putting sensitive prompts, credentials, or customer information into telemetry.

Retry Observability

Retries should be visible.

Consider:

Tool Call
   ↓
Failure
   ↓
Retry 1
   ↓
Failure
   ↓
Retry 2
   ↓
Success

Record the retry attempt:

logger.LogWarning(
    "Tool {ToolName} retry attempt {Attempt}",
    toolName,
    attempt);

Useful metrics include:

Retry Count
Retry Success Rate
Final Failure Rate
Total Retry Delay

A rising retry rate can indicate an underlying service problem even if the agent's final success rate remains high.

Tracking Agent Errors by Category

Not all failures are equivalent.

Define categories such as:

ModelError
ToolError
Timeout
AuthenticationError
ValidationError
RateLimit
ApplicationError
Cancellation

A simple enum can help:

public enum AgentErrorType
{
    ModelError,
    ToolError,
    Timeout,
    AuthenticationError,
    ValidationError,
    RateLimit,
    ApplicationError,
    Cancellation
}

Categorization makes dashboards much more useful.

Instead of:

Failures: 1,240

you can see:

Tool Error:          620
Timeout:             310
Rate Limit:          180
Model Error:          90
Other:                40

Do Not Log Sensitive Prompts by Default

Agent prompts can contain confidential information.

They may include:

Customer information
Internal documentation
Source code
Access details
Business data

Therefore, avoid blindly logging the complete prompt:

logger.LogInformation(
    "Prompt: {Prompt}",
    prompt);

Instead, log metadata:

logger.LogInformation(
    "Model request started. " +
    "Model={Model}, ExecutionId={ExecutionId}",
    modelName,
    executionId);

If detailed prompt logging is genuinely required for debugging, apply strict access controls, retention rules, redaction, and data-handling policies.

Redacting Sensitive Data

If telemetry needs selected request information, redact sensitive fields before recording them.

For example:

public static string RedactEmail(
    string email)
{
    var at = email.IndexOf('@');

    if (at <= 1)
    {
        return "***";
    }

    return email[0] + "***" + email[at..];
}

Do not rely exclusively on manual redaction for complex data.

A better production approach is to minimize the sensitive data entering telemetry in the first place.

Tracking Agent Cost

If reliable usage and pricing information are available, cost can be calculated separately from token telemetry.

For example:

public sealed record UsageCost(
    long InputTokens,
    long OutputTokens,
    decimal? ReportedCost);

Avoid hard-coding an assumed token price unless it corresponds to the exact model and billing arrangement being evaluated.

Usage and billing should remain separate concepts:

Token Usage
    ↓
Usage Analytics

Provider Billing
    ↓
Actual Cost

Create an Agent Metrics Model

A consolidated model can simplify reporting:

public sealed record AgentExecutionMetrics(
    string ExecutionId,
    string AgentName,
    TimeSpan Duration,
    int ModelCalls,
    int ToolCalls,
    long InputTokens,
    long OutputTokens,
    int RetryCount,
    bool Succeeded);

This model can be populated after an execution completes.

It provides a useful summary without requiring the dashboard to reconstruct the entire trace.

Example Agent Execution

A simplified service might look like:

public async Task<string> ExecuteAsync(
    string request,
    CancellationToken cancellationToken)
{
    var executionId = Guid.NewGuid();

    using var activity =
        AgentTelemetry.Source.StartActivity(
            "agent.execute");

    activity?.SetTag(
        "agent.execution_id",
        executionId.ToString());

    var stopwatch = Stopwatch.StartNew();

    try
    {
        logger.LogInformation(
            "Agent execution started: {ExecutionId}",
            executionId);

        var result =
            await RunAgentAsync(
                request,
                cancellationToken);

        stopwatch.Stop();

        logger.LogInformation(
            "Agent execution completed. " +
            "ExecutionId={ExecutionId}, " +
            "DurationMs={DurationMs}",
            executionId,
            stopwatch.Elapsed.TotalMilliseconds);

        return result;
    }
    catch (OperationCanceledException)
    {
        stopwatch.Stop();

        logger.LogWarning(
            "Agent execution cancelled: {ExecutionId}",
            executionId);

        throw;
    }
    catch (Exception ex)
    {
        stopwatch.Stop();

        activity?.SetTag(
            "agent.success",
            false);

        logger.LogError(
            ex,
            "Agent execution failed: {ExecutionId}",
            executionId);

        throw;
    }
}

The important design principle is that telemetry surrounds the agent workflow without becoming part of the business logic.

Observability Dashboard

A production dashboard should answer operational questions quickly.

Traffic

Agent Executions
Successful Executions
Failed Executions

Latency

Average Duration
P95 Duration
P99 Duration

Model Usage

Model
Calls
Input Tokens
Output Tokens

Tool Usage

Tool
Calls
Failures
Average Duration

Reliability

Failure Rate
Retry Rate
Timeout Rate

Avoid using averages alone.

Tail latency such as P95 and P99 can reveal slow executions that an average hides.

Common Mistakes

Logging Only the Final Response

This makes multi-step failures difficult to diagnose.

Logging Complete Prompts

Prompts can contain sensitive information.

Measuring Only Total Latency

Without child-operation latency, identifying bottlenecks becomes difficult.

Ignoring Tool Failures

A successful final response can hide repeated tool failures and retries.

Treating All Errors the Same

Categorized failures are much easier to investigate.

Assuming Token Usage Is Always Available

Some providers or SDK paths may not expose complete usage information.

Creating Excessive Telemetry

Recording every piece of agent context can increase storage, cost, and privacy risk.

Best Practices

Generate a unique execution ID for every agent workflow.
Use structured logging.
Represent model calls and tool calls as observable operations.
Track total and component-level latency.
Capture token usage when reliably available.
Categorize failures.
Record retry attempts.
Use distributed tracing for multi-service workflows.
Avoid logging sensitive prompts and responses by default.
Apply appropriate retention and access controls.
Monitor tail latency such as P95 and P99.
Separate usage analytics from billing calculations.
Keep telemetry concerns separate from business logic.
Test observability during failure scenarios, not only successful executions.

Advantages and Disadvantages

Advantages

Makes multi-step agent behavior easier to understand
Reduces debugging time
Helps identify slow tools and model calls
Provides visibility into failures and retries
Supports usage analysis
Enables production monitoring
Helps identify reliability trends

Disadvantages

Additional telemetry increases implementation complexity
Detailed traces can increase storage requirements
Agent traces may contain sensitive information
Token information may not always be available
Excessive instrumentation can create unnecessary noise
AI workflows can generate significantly more events than traditional requests

Troubleshooting Observability Problems

If an agent trace is incomplete:

Verify that the ActivitySource is registered with the telemetry pipeline.
Check that child activities are created within the parent execution.
Confirm that the required instrumentation is enabled.
Verify that structured logging properties are preserved.
Check whether exceptions are being swallowed.
Confirm that cancellation is recorded correctly.
Verify token usage is actually returned by the model client.
Check whether tool failures are recorded before retries.
Review telemetry sampling configuration.
Confirm that sensitive fields are being removed or redacted.

If traces show the agent execution but not individual model or tool operations, inspect the instrumentation boundaries first.

A Production-Oriented Architecture

A practical .NET architecture can look like:

                  AI Agent
                     |
             +-------+-------+
             |               |
             v               v
        Model Client      Tool Layer
             |               |
             v               v
        Model Calls       Tool Calls
             |               |
             +-------+-------+
                     |
                     v
              Telemetry Layer
                     |
          +----------+----------+
          |          |          |
          v          v          v
        Logs      Metrics     Traces
          |          |          |
          +----------+----------+
                     |
                     v
               Observability
                Dashboard

This separation allows developers to improve telemetry without changing the agent's core business behavior.

Conclusion

AI agent observability requires more than traditional request logging because an agent can perform many operations before returning a result. A production system should make those operations visible while carefully controlling the amount of sensitive information captured.

The most useful foundation is to track execution IDs, model calls, tool calls, token usage, latency, retries, and categorized failures.

OpenTelemetry-compatible tracing can connect these individual operations into a single execution trace, while structured logs and metrics provide searchable operational data.

The goal is not to record everything an agent sees or thinks. The goal is to record enough reliable information to answer a practical production question:

What happened during this agent execution, how long did it take, where did it fail, and what resources did it consume?

With that visibility, development teams can troubleshoot agent failures more effectively, identify performance bottlenecks, monitor reliability, and make informed decisions about how AI workloads operate in production.