AI agents introduce a different observability challenge compared with traditional applications.
A normal application may have a relatively predictable request path:
User
|
v
API
|
v
Database
|
v
Response
An AI agent can execute a much longer workflow:
User
|
v
Agent
|
+--> LLM
|
+--> Tool
|
+--> Database
|
+--> External API
|
+--> Another model
|
v
Final Response
When something becomes slow or fails, developers need to determine which part of the workflow caused the problem.
Was the model slow?
Did a tool timeout?
Was the database query expensive?
Did the agent retry an operation?
Was a cloud service unavailable?
This is where centralized observability becomes important.
Amazon CloudWatch provides monitoring and observability capabilities across AWS workloads. CloudWatch cross-account observability, often associated with CloudWatch's broader observability capabilities, can help teams bring telemetry from multiple AWS accounts and applications into a unified monitoring experience.
For AI agents, the same principle can be extended across the infrastructure supporting the agent: applications, compute resources, APIs, databases, queues, and model-related services.
This article explains how to design an observability architecture for AI agents using Amazon CloudWatch and how to connect agent-level telemetry with traditional cloud monitoring.
Why AI Agent Observability Is Different
Traditional application monitoring often focuses on metrics such as:
CPU utilization
Memory usage
Request latency
HTTP errors
Database latency
These metrics remain important for AI applications, but they are not enough.
An AI agent introduces additional dimensions:
Model latency
Token consumption
Tool execution time
Number of agent steps
Tool failures
Retry count
Prompt size
Output size
Workflow completion rate
Human approval time
A useful agent trace might look like:
Agent Request
|
+---- LLM Call: 420 ms
|
+---- Search Tool: 180 ms
|
+---- Database Query: 75 ms
|
+---- LLM Call: 650 ms
|
+---- Email Tool: 230 ms
|
v
Completed
Without this information, developers may see only:
POST /api/agent -> 1.8 seconds
That does not explain why the request took 1.8 seconds.
A Unified CloudWatch Architecture
A practical architecture can connect application telemetry with infrastructure telemetry:
User
|
v
+---------------+
| Agent App |
+---------------+
|
+--------+--------+
| | |
v v v
LLM Tools Database
| | |
+--------+--------+
|
v
CloudWatch
/ | \
/ | \
Logs Metrics Traces
\ | /
\ | /
+----+-----+
|
v
Dashboards
& Alerts
The goal is to correlate application-level agent behavior with infrastructure-level health.
What Should Be Monitored?
A production AI agent should expose telemetry at several levels.
Layer | Useful Signals |
|---|---|
Application | Request count, latency, errors |
Agent | Steps, completion rate, retries |
Model | Latency, token usage, failures |
Tools | Execution time, errors, timeouts |
Database | Query latency, connections, errors |
Infrastructure | CPU, memory, network |
Security | Authentication failures, denied operations |
Cost | Requests, tokens, compute consumption |
This layered approach makes troubleshooting much easier.
CloudWatch Logs for Agent Activity
Logs are useful for capturing discrete events.
For example:
logger.LogInformation(
"Agent started. RequestId={RequestId}, Agent={AgentName}",
requestId,
agentName);
When a tool starts:
logger.LogInformation(
"Agent tool started. RequestId={RequestId}, Tool={ToolName}",
requestId,
toolName);
And when it completes:
logger.LogInformation(
"Agent tool completed. RequestId={RequestId}, Tool={ToolName}, DurationMs={DurationMs}",
requestId,
toolName,
stopwatch.ElapsedMilliseconds);
Structured logging is preferable to unstructured messages because fields such as RequestId, ToolName, and DurationMs can later be queried and aggregated.
A useful log event might contain:
{
"requestId": "req-4821",
"agent": "OrderAssistant",
"tool": "OrderSearch",
"durationMs": 182,
"status": "success"
}
Avoid putting prompts, credentials, access tokens, or sensitive customer information into logs.
Metrics Are Better for Trends
Logs explain individual events.
Metrics help answer questions about the system as a whole.
For an AI agent, useful custom metrics include:
AgentRequests
AgentFailures
AgentDuration
ToolDuration
ToolFailures
AgentRetries
TokensConsumed
For example:
AgentRequests = 12,400
AgentFailures = 183
Failure Rate = 1.48%
The exact calculation can be performed by the monitoring platform or application analytics layer.
The important part is to emit consistent measurements.
Measuring Agent Step Duration
Consider an agent that executes five steps:
Step 1: Understand request
Step 2: Search documents
Step 3: Query database
Step 4: Call external API
Step 5: Generate response
The application can record duration for each step.
var stopwatch = Stopwatch.StartNew();
try
{
await ExecuteToolAsync(cancellationToken);
}
finally
{
stopwatch.Stop();
logger.LogInformation(
"Tool execution completed. Tool={Tool}, DurationMs={DurationMs}",
toolName,
stopwatch.ElapsedMilliseconds);
}
This makes it possible to identify the slowest part of the workflow.
For example:
Document Search 120 ms
Database Query 80 ms
External API 950 ms
LLM Generation 700 ms
The external API would deserve investigation before optimizing the database query.
Distributed Tracing
Logs and metrics become significantly more useful when requests can be correlated.
Consider:
Request ID: 7f29
Agent
|
+-- LLM Call
|
+-- Search Tool
| |
| +-- Database
|
+-- Email Tool
|
+-- Email API
Distributed tracing can represent these operations as related spans.
Conceptually:
Agent Request
├── Model Call
├── Search Tool
│ └── Database Query
└── Email Tool
└── External API
This allows engineers to move from an overall slow request to the specific operation responsible for the delay.
For .NET applications, OpenTelemetry can be used to instrument application code and export telemetry to supported observability systems.
Correlation IDs
A simple correlation strategy is essential.
Generate or propagate a request identifier:
var requestId = Activity.Current?.TraceId.ToString()
?? Guid.NewGuid().ToString();
Then include it in logs and relevant application events.
For example:
RequestId: 9e13
Agent started
Tool started: SearchOrders
Tool completed: SearchOrders
LLM call started
LLM call completed
Agent completed
Now a developer can search for one identifier rather than trying to reconstruct a workflow from thousands of unrelated log messages.
Monitoring Tool Calls
Tools are one of the most important areas to monitor because agents can invoke them dynamically.
Suppose an agent has:
SearchOrders
GetCustomer
GetInventory
SendEmail
CreateTicket
Track each tool independently.
Tool | Calls | Errors | Avg. Duration |
|---|---|---|---|
SearchOrders | 8,240 | 94 | 120 ms |
GetCustomer | 7,900 | 31 | 65 ms |
GetInventory | 4,120 | 83 | 310 ms |
SendEmail | 1,240 | 17 | 240 ms |
This immediately highlights unusual behavior.
For example, a sudden increase in GetInventory latency may explain an increase in overall agent latency.
Monitoring Agent Retries
Agents can produce repeated tool calls because of failures, transient errors, or workflow behavior.
Track retries explicitly.
Agent Requests
|
+---- First attempt
|
+---- Retry
|
+---- Retry
|
v
Final Result
A high retry rate can increase:
Latency
Infrastructure usage
API traffic
Model calls
Cost
A useful metric is:
Retry Rate =
Number of Retried Operations
/
Total Operations
Do not assume retries are always bad. They can be valuable for transient failures. The important issue is whether retry behavior is controlled and justified.
Monitoring Model Calls
AI agents may invoke a model multiple times during one workflow.
For example:
User Request
|
v
LLM Call 1 -> Planning
|
v
Tool
|
v
LLM Call 2 -> Interpretation
|
v
Tool
|
v
LLM Call 3 -> Final Response
A single user request therefore does not necessarily equal one model request.
Monitor:
Number of model calls per workflow
Model latency
Input tokens
Output tokens
Errors
Timeouts
Model selection
This can reveal unexpectedly expensive workflows.
Connecting Logs, Metrics, and Traces
The three major observability signals have different purposes.
Logs
|
+--> What happened?
Metrics
|
+--> How often is it happening?
Traces
|
+--> Where did it happen in the request?
For example:
Metric
Agent p95 latency increased from 900 ms to 2.1 s.
Trace
External API span increased from 150 ms to 1.3 s.
Log
External API timeout occurred for endpoint X.
Together, these signals provide a much clearer diagnosis than any one of them alone.
CloudWatch Dashboards for AI Agents
A dedicated dashboard can provide a high-level view of the agent platform.
For example:
+------------------------------------------------+
| AI Agent Operations |
+------------------------------------------------+
| Requests | Error Rate | p95 Latency |
| 125K | 1.2% | 1.8 sec |
+------------------------------------------------+
| Tool Failures | Retries | Active Workflows |
| 842 | 2.4% | 1,240 |
+------------------------------------------------+
| Agent Latency |
| Requests / Latency over time |
+------------------------------------------------+
| Tool Performance |
| Search | Database | API | Email |
+------------------------------------------------+
The dashboard should focus on operational questions rather than showing every available metric.
Alerts
Monitoring becomes useful when important changes trigger action.
Examples include:
Agent error rate > threshold
Tool latency > threshold
Model timeout rate > threshold
Database errors > threshold
Agent retry rate > threshold
For example:
IF
AgentErrorRate > 5%
THEN
Create operational alert
Thresholds should be based on normal workload behavior.
A threshold that is too sensitive can generate alert fatigue.
Multi-Account Observability
Large AWS environments often separate workloads across multiple accounts.
For example:
Production Account
|
+---- Agent API
+---- Database
AI Account
|
+---- Model Infrastructure
Data Account
|
+---- Analytics
Security Account
|
+---- Monitoring
Centralized observability can help teams inspect telemetry across these environments without requiring engineers to manually switch between every account.
This becomes particularly useful when an agent spans multiple AWS services and accounts.
For example:
Agent API
|
v
Account A
|
+----> Account B: Model
|
+----> Account C: Data
|
+----> Account D: External Integration
A cross-account observability strategy can provide a more complete operational picture.
Security and Privacy
AI telemetry can contain sensitive information.
A careless logging strategy might capture:
User Prompt
Customer Data
Access Token
API Key
Database Result
This creates a significant security risk.
Instead, logs should contain operational metadata where possible:
{
"requestId": "req-4821",
"agent": "OrderAssistant",
"tool": "OrderSearch",
"status": "success",
"durationMs": 182
}
Avoid logging full prompts or tool responses unless there is a specific, controlled requirement.
When sensitive information must be captured for troubleshooting, apply appropriate redaction, access controls, retention policies, and encryption.
Cost Observability
Observability should also include cost-related signals.
For an AI agent, cost can increase because of:
More model calls
Larger prompts
Longer outputs
Excessive retries
Inefficient tool workflows
Increased infrastructure usage
Consider tracking:
Requests per workflow
Model calls per request
Tokens per request
Retries per request
Tool calls per request
A sudden increase in average model calls may indicate an agent workflow regression.
For example:
Before:
1.8 model calls / request
After:
4.7 model calls / request
That change deserves investigation even if the application still appears functional.
A Practical .NET Observability Pattern
A reusable telemetry service can keep instrumentation consistent.
public interface IAgentTelemetry
{
void AgentStarted(string requestId, string agent);
void ToolStarted(
string requestId,
string tool);
void ToolCompleted(
string requestId,
string tool,
long durationMs);
void AgentCompleted(
string requestId,
long durationMs);
void AgentFailed(
string requestId,
Exception exception);
}
The agent service can then use it:
var stopwatch = Stopwatch.StartNew();
telemetry.AgentStarted(
requestId,
"OrderAssistant");
try
{
await ExecuteWorkflowAsync(
request,
cancellationToken);
telemetry.AgentCompleted(
requestId,
stopwatch.ElapsedMilliseconds);
}
catch (Exception ex)
{
telemetry.AgentFailed(
requestId,
ex);
throw;
}
This keeps observability concerns from spreading throughout every component.
Common Mistakes
Logging the Entire Prompt
This can expose sensitive information and increase log volume.
Log metadata instead unless the actual prompt is specifically required for controlled debugging.
Monitoring Only Infrastructure
CPU and memory can look healthy while the agent is producing slow or incorrect workflows.
Monitor agent behavior as well as infrastructure.
Ignoring Tool Latency
A fast model does not help if the agent spends several seconds waiting for an external API.
Measure each tool separately.
Creating Too Many Metrics
Every metric should have a purpose.
Avoid creating hundreds of metrics that nobody uses.
Start with signals that support operational decisions.
No Correlation Between Signals
A metric saying that latency increased is useful.
A trace and correlated log explaining why it increased are much more useful.
Alerting on Everything
Not every error deserves an immediate page.
Separate informational events from actionable operational failures.
Best Practices
When monitoring AI agents with CloudWatch and related AWS observability capabilities:
Instrument the entire agent workflow.
Track model, tool, application, and infrastructure latency separately.
Use correlation IDs or trace context consistently.
Prefer structured logs.
Monitor agent-specific metrics in addition to infrastructure metrics.
Track retries and repeated model calls.
Monitor tool-level failure rates.
Measure p50, p95, and p99 latency.
Create dashboards around operational questions.
Keep sensitive prompt and tool data out of logs where possible.
Use centralized observability for multi-account environments.
Create alerts around actionable thresholds.
Monitor token and workflow behavior for cost anomalies.
Use distributed tracing to connect application and infrastructure behavior.
Conclusion
AI agents make cloud observability more complex because one user request can trigger many model calls, tools, APIs, databases, and infrastructure components.
Amazon CloudWatch can provide an important observability layer for AWS-based AI applications by bringing together logs, metrics, dashboards, alarms, and related telemetry. Cross-account observability can be particularly useful when production systems are distributed across multiple AWS accounts.
The key is to monitor the agent as a workflow rather than treating it as a single API request.
A useful mental model is:
User Request
|
v
Agent Workflow
|
+---- Model
|
+---- Tool
|
+---- Database
|
+---- API
|
v
Final Response
Every important stage should be observable.
When logs, metrics, and traces are connected through consistent request or trace identifiers, developers can move from a high-level symptom such as increased latency to the specific model call, tool, database query, or infrastructure component responsible for the problem.
Summary
Monitoring AI agents requires more than traditional cloud metrics. Teams need visibility into agent workflows, model calls, tool execution, retries, latency, errors, and resource usage.
CloudWatch can serve as a central observability layer for AWS workloads, while structured application telemetry connects agent behavior with the underlying infrastructure. A well-designed monitoring strategy should provide enough information to answer three questions quickly:
What happened?
How often is it happening?
Where is the problem occurring?
Once those questions can be answered reliably, AI agent systems become significantly easier to operate, troubleshoot, secure, and optimize in production.
Join the conversation! Your thoughts help the community grow.