AI agents can perform increasingly complex tasks, from retrieving information and calling tools to generating code and executing multi-step workflows. However, evaluating an agent only by checking its final response is not enough for production systems.
An agent can produce a convincing answer while using the wrong tool, violating a business rule, omitting an important step, or making an unsafe decision.
This creates a fundamental engineering challenge: How can teams verify AI-driven behavior when part of the system is inherently non-deterministic?
A practical solution is to combine three verification layers:
Deterministic automated tests
AI-based evaluation
Human review for high-impact or ambiguous cases
This article presents a practical verification architecture for .NET-based agentic applications and explains how these layers can work together.
Why AI Agents Need Multiple Verification Layers
Traditional software generally follows a predictable execution model:
Input
↓
Application Logic
↓
Expected Output
AI agents introduce variability:
User Request
↓
AI Agent
|
+---- Model Decision
|
+---- Tool Selection
|
+---- External Data
|
+---- Additional Reasoning
|
↓
Final Result
The same request may not always produce exactly the same execution path.
Therefore, verification should focus on invariants and outcomes rather than requiring every internal step to be identical.
For example, an order-management agent might be allowed to use different reasoning paths, but these conditions should remain true:
Customer must be authorized
Order must belong to customer
Cancellation must be allowed
Cancellation must be recorded
These are deterministic business requirements even if the agent's reasoning is not.
The Three-Layer Verification Model
A robust architecture can look like this:
AI Agent
|
+------------+------------+
| | |
v v v
Deterministic AI Evaluator Human Review
Checks
| | |
+------------+------------+
|
v
Verification Result
Each layer has a different responsibility.
| Layer | Best suited for |
|---|
| Deterministic tests | Business rules and security |
| AI evaluator | Open-ended quality and relevance |
| Human review | Ambiguous or high-impact decisions |
The goal is not to make one layer replace the others.
Deterministic Verification
Deterministic tests should protect rules that must never be violated.
For example:
public sealed record Order(
string Id,
string CustomerId,
string Status);
An authorization rule can be tested independently:
public static bool CanCancel(
string customerId,
Order order)
{
return order.CustomerId == customerId
&& order.Status == "Pending";
}
Then:
[Fact]
public void CustomerCannotCancelAnotherCustomersOrder()
{
var order = new Order(
"ORD-1001",
"customer-2",
"Pending");
var result = CanCancel(
"customer-1",
order);
Assert.False(result);
}
The AI agent should never be responsible for enforcing this rule by itself.
Protect Business Invariants
Business invariants are particularly valuable for agent verification.
Suppose an agent is responsible for issuing refunds.
The following conditions might be mandatory:
Refund amount <= eligible amount
Customer is authorized
Order exists
Order is refundable
Refund operation is recorded
Represent these rules independently:
public sealed record RefundRequest(
string CustomerId,
string OrderId,
decimal Amount);
Then validate:
public static bool IsValidRefund(
RefundRequest request,
Order order)
{
return order.CustomerId == request.CustomerId
&& order.Status == "Completed"
&& request.Amount > 0
&& request.Amount <= order.RefundableAmount;
}
The agent can decide how to handle the user's request, but the application controls whether the actual operation is permitted.
Verify Tool Calls
The final response is only one part of an agent execution.
Consider:
User:
"Cancel order ORD-1001."
Agent:
1. Get order
2. Check status
3. Cancel order
4. Confirm cancellation
The harness should capture these actions.
A simple model can represent a tool invocation:
public sealed record ToolCall(
string Name,
IReadOnlyDictionary<string, object?> Arguments);
Then verify the expected behavior:
[Fact]
public void CancellationUsesExpectedTool()
{
var call = new ToolCall(
"CancelOrder",
new Dictionary<string, object?>
{
["orderId"] = "ORD-1001"
});
Assert.Equal(
"CancelOrder",
call.Name);
Assert.Equal(
"ORD-1001",
call.Arguments["orderId"]);
}
This catches errors that may not be visible in the final answer.
Verify Tool Arguments
A correct tool with incorrect arguments can still cause a serious problem.
For example:
Tool: CancelOrder
Expected:
orderId = ORD-1001
Actual:
orderId = ORD-1002
The agent might produce a perfectly reasonable confirmation message while performing the wrong operation.
For state-changing tools, argument validation should therefore be stricter than for informational tools.
Classify Tool Risk
A useful approach is to classify tools by impact.
Read
↓
Low Risk
Write
↓
Medium Risk
Financial / Destructive
↓
High Risk
For example:
| Tool | Risk | Verification |
|---|
| SearchProducts | Low | Input validation |
| GetCustomer | Low | Authorization |
| UpdateProfile | Medium | Authorization + validation |
| CancelOrder | High | Authorization + confirmation |
| IssueRefund | Critical | Strong authorization + business rules |
The exact classification depends on the application.
AI-Based Evaluation
Deterministic checks cannot answer every question.
Consider:
"Was the response clear and relevant?"
There may not be a single exact expected string.
An AI evaluator can assess characteristics such as:
Relevance
Completeness
Clarity
Grounding
Instruction adherence
Response quality
A structured evaluation result can be represented as:
public sealed record AiEvaluation(
double Score,
bool Passed,
string Reason);
For example:
{
"score": 4,
"passed": true,
"reason": "The response directly answered the request and used the provided customer information."
}
The evaluator should use a defined rubric rather than an unrestricted question such as:
"Is this good?"
Designing an Evaluation Rubric
A rubric makes AI-based evaluation more consistent.
For example:
Score 1:
Incorrect or irrelevant
Score 2:
Partially useful but major omissions
Score 3:
Acceptable response
Score 4:
Correct and complete
Score 5:
Correct, complete, clear, and appropriately grounded
A structured evaluator prompt can then request:
Evaluate the response using the supplied rubric.
Check:
- Factual correctness
- Relevance
- Completeness
- Grounding
- Instruction adherence
Return:
- Score
- Pass/fail
- Short reason
The evaluator's own variability should still be considered.
Do Not Use AI Evaluation for Security Authorization
An AI evaluator should not determine whether an operation is authorized.
For example, avoid:
AI:
"Should this user be allowed to issue a refund?"
Instead:
Authorization Service
↓
Allowed / Denied
↓
Agent
Security and business authorization should remain deterministic.
AI evaluation can assess whether the agent's response correctly communicated the result, but not whether the user actually has permission.
Human Review
Some scenarios are too important or ambiguous for automated evaluation alone.
Human review is appropriate for:
A review queue can be represented as:
Agent Execution
↓
Automated Verification
|
+---- Clearly Passed → Accept
|
+---- Clearly Failed → Reject
|
+---- Uncertain → Human Review
This reduces the amount of manual review while preserving human oversight where it matters.
Confidence-Based Routing
Suppose an evaluator produces:
Score: 4.8
Confidence: High
The case may be automatically accepted.
If it produces:
Score: 2.7
Confidence: Low
send it to review.
A model can represent this:
public sealed record VerificationDecision(
string Status,
double Confidence);
Then:
public static string Route(
VerificationDecision decision)
{
if (decision.Confidence < 0.70)
{
return "HumanReview";
}
return decision.Status;
}
The threshold should be determined through validation rather than chosen arbitrarily for a production system.
Evaluation Datasets
Create a version-controlled collection of representative scenarios.
[
{
"id": "order-001",
"input": "Where is my order?",
"expectedBehavior": "Return current order status."
},
{
"id": "order-002",
"input": "Cancel my order.",
"expectedBehavior": "Verify authorization before cancellation."
}
]
Each evaluation case can include:
Input
Expected behavior
Risk level
Required tools
Forbidden actions
Expected business rules
Evaluation criteria
This transforms testing from an ad hoc process into a repeatable verification suite.
Testing Non-Deterministic Responses
Do not always compare generated text exactly.
This is fragile:
Assert.Equal(
"Your order has been cancelled.",
response);
A semantically equivalent response could be:
"Order ORD-1001 has been successfully cancelled."
Instead, test deterministic properties:
Assert.Contains(
"cancelled",
response,
StringComparison.OrdinalIgnoreCase);
For more complex responses, use structured output or an evaluator.
The verification target should be the behavior that matters, not unnecessary wording details.
Testing Grounded Responses
If an agent uses retrieved information, verify that the response is grounded in the available data.
For example:
Retrieved Order:
Status = Shipped
Agent Response:
"Your order is currently shipped."
But:
Retrieved Order:
Status = Shipped
Agent Response:
"Your order will arrive tomorrow at 2 PM."
The second statement introduces information that was not present in the supplied data.
A grounding evaluator can flag unsupported claims.
Verification Pipeline
A complete evaluation can follow:
Test Case
↓
Agent Execution
↓
Capture Trace
↓
Validate Tool Calls
↓
Validate Business Rules
↓
Validate Security Rules
↓
Evaluate Response
↓
Determine Confidence
↓
Human Review if Required
↓
Final Result
This creates multiple opportunities to catch failures.
Example Verification Orchestrator
A simplified C# orchestrator could look like:
public sealed class AgentVerifier
{
public async Task<VerificationResult> VerifyAsync(
EvaluationCase testCase,
CancellationToken cancellationToken)
{
var execution =
await RunAgentAsync(
testCase,
cancellationToken);
var deterministicResult =
ValidateDeterministicRules(execution);
if (!deterministicResult.Passed)
{
return VerificationResult.Failed(
deterministicResult.Reason);
}
var aiResult =
await EvaluateResponseAsync(
execution,
cancellationToken);
if (aiResult.Confidence < 0.70)
{
return VerificationResult.NeedsReview(
"Evaluation confidence is low.");
}
return aiResult.Passed
? VerificationResult.Passed()
: VerificationResult.Failed(
aiResult.Reason);
}
}
The example intentionally keeps the implementation simple. In a production system, the verification stages should be independently testable and observable.
Observability and Verification
Verification becomes significantly more useful when execution traces are available.
A trace might show:
Agent Request
|
+---- Model Call
|
+---- GetCustomer
|
+---- GetOrder
|
+---- CancelOrder
|
+---- Model Call
|
+---- Final Response
If the final result fails evaluation, developers can inspect the execution path rather than guessing what happened.
OpenTelemetry can be used to instrument application-level operations and propagate trace context through supported dependencies.
Verification Results
Store structured results instead of only writing pass/fail messages to a log.
public sealed record VerificationResult(
string TestId,
bool Passed,
string Status,
string? Reason,
TimeSpan Duration);
Possible statuses include:
Passed
Failed
NeedsReview
Blocked
Error
This provides a useful distinction between an actual agent failure and an infrastructure problem.
Common Mistakes
Testing Only the Final Response
The agent may have taken an unsafe path while producing a convincing answer.
Allowing AI to Enforce Authorization
Authorization should remain deterministic.
Comparing Exact Text
Natural-language responses can vary while remaining correct.
Ignoring Tool Arguments
Calling the correct tool with the wrong identifier can still cause a serious failure.
Sending Every Case to Humans
Manual review does not scale.
Automate deterministic checks and route only uncertain or high-risk cases to people.
Using Only One Evaluator
For critical systems, combine multiple verification mechanisms instead of relying entirely on an AI judge.
Best Practices
Separate deterministic rules from AI-based evaluation.
Keep authorization and security controls deterministic.
Validate tool selection and arguments.
Use structured evaluation datasets.
Avoid exact text matching for open-ended responses.
Classify tools by risk.
Route uncertain cases to human reviewers.
Capture execution traces.
Store structured verification results.
Version evaluation datasets and rubrics.
Test both successful and adversarial scenarios.
Monitor evaluation failures over time.
Re-run regression evaluations after model or prompt changes.
Keep sensitive data out of unnecessary traces and evaluation records.
Advantages and Disadvantages
Advantages
Combines automated and human verification
Protects deterministic business rules
Detects incorrect tool usage
Supports open-ended response evaluation
Scales better than manual review alone
Provides structured regression testing
Improves visibility into agent failures
Disadvantages
More complex than traditional unit testing
AI evaluators can introduce variability
Human review is still required for some cases
Evaluation datasets require maintenance
Trace data can contain sensitive information
Multiple verification layers increase implementation effort
Troubleshooting Verification Failures
When an evaluation fails, inspect the layers in order:
Verify the test input.
Check the agent execution.
Inspect tool calls.
Validate tool arguments.
Check deterministic business rules.
Review authorization decisions.
Inspect retrieved data.
Evaluate the final response.
Review evaluator confidence.
Determine whether human review is necessary.
This avoids changing the AI prompt when the actual problem may be an application bug or incorrect test expectation.
A Practical Enterprise Architecture
A production-oriented verification platform can be organized as:
Agent Application
|
v
Execution Harness
|
+--------------+--------------+
| | |
v v v
Trace Capture Rule Engine AI Evaluator
| | |
+--------------+--------------+
|
v
Decision Engine
|
+----------+----------+
| |
v v
Auto Accept Human Review
The decision engine can combine:
This creates a scalable verification model without pretending that every AI decision can be reduced to a single deterministic assertion.
Conclusion
AI agents require a different verification strategy from traditional deterministic software. The goal is not to force every model response to be identical. Instead, teams should define the behaviors and invariants that must remain reliable regardless of how the agent reaches its conclusion.
A strong verification framework combines deterministic tests, AI-based evaluation, execution tracing, risk-aware tool validation, and human review.
Deterministic code should enforce authorization, business rules, and other critical constraints. AI evaluators can assess open-ended qualities such as relevance and completeness. Human reviewers should handle ambiguous or high-impact cases.
The resulting architecture creates a practical balance between automation and control:
Deterministic Rules
+
AI Evaluation
+
Human Oversight
↓
Reliable Agent Verification
This approach allows organizations to adopt agentic AI while maintaining the engineering discipline required for systems that operate on real data, invoke real tools, and participate in business-critical workflows.