Agentic AI Verification Framework: Combining AI, Humans, and Deterministic Tests

AI agents can perform increasingly complex tasks, from retrieving information and calling tools to generating code and executing multi-step workflows. However, evaluating an agent only by checking its final response is not enough for production systems.

An agent can produce a convincing answer while using the wrong tool, violating a business rule, omitting an important step, or making an unsafe decision.

This creates a fundamental engineering challenge: How can teams verify AI-driven behavior when part of the system is inherently non-deterministic?

A practical solution is to combine three verification layers:

Deterministic automated tests
AI-based evaluation
Human review for high-impact or ambiguous cases

This article presents a practical verification architecture for .NET-based agentic applications and explains how these layers can work together.

Why AI Agents Need Multiple Verification Layers

Traditional software generally follows a predictable execution model:

Input
  ↓
Application Logic
  ↓
Expected Output

AI agents introduce variability:

User Request
     ↓
AI Agent
     |
     +---- Model Decision
     |
     +---- Tool Selection
     |
     +---- External Data
     |
     +---- Additional Reasoning
     |
     ↓
Final Result

The same request may not always produce exactly the same execution path.

Therefore, verification should focus on invariants and outcomes rather than requiring every internal step to be identical.

For example, an order-management agent might be allowed to use different reasoning paths, but these conditions should remain true:

Customer must be authorized
Order must belong to customer
Cancellation must be allowed
Cancellation must be recorded

These are deterministic business requirements even if the agent's reasoning is not.

The Three-Layer Verification Model

A robust architecture can look like this:

                    AI Agent
                       |
          +------------+------------+
          |            |            |
          v            v            v
   Deterministic   AI Evaluator   Human Review
      Checks
          |            |            |
          +------------+------------+
                       |
                       v
                Verification Result

Each layer has a different responsibility.

Layer	Best suited for
Deterministic tests	Business rules and security
AI evaluator	Open-ended quality and relevance
Human review	Ambiguous or high-impact decisions

The goal is not to make one layer replace the others.

Deterministic Verification

Deterministic tests should protect rules that must never be violated.

For example:

public sealed record Order(
    string Id,
    string CustomerId,
    string Status);

An authorization rule can be tested independently:

public static bool CanCancel(
    string customerId,
    Order order)
{
    return order.CustomerId == customerId
        && order.Status == "Pending";
}

Then:

[Fact]
public void CustomerCannotCancelAnotherCustomersOrder()
{
    var order = new Order(
        "ORD-1001",
        "customer-2",
        "Pending");

    var result = CanCancel(
        "customer-1",
        order);

    Assert.False(result);
}

The AI agent should never be responsible for enforcing this rule by itself.

Protect Business Invariants

Business invariants are particularly valuable for agent verification.

Suppose an agent is responsible for issuing refunds.

The following conditions might be mandatory:

Refund amount <= eligible amount
Customer is authorized
Order exists
Order is refundable
Refund operation is recorded

Represent these rules independently:

public sealed record RefundRequest(
    string CustomerId,
    string OrderId,
    decimal Amount);

Then validate:

public static bool IsValidRefund(
    RefundRequest request,
    Order order)
{
    return order.CustomerId == request.CustomerId
        && order.Status == "Completed"
        && request.Amount > 0
        && request.Amount <= order.RefundableAmount;
}

The agent can decide how to handle the user's request, but the application controls whether the actual operation is permitted.

Verify Tool Calls

The final response is only one part of an agent execution.

Consider:

User:
"Cancel order ORD-1001."

Agent:
1. Get order
2. Check status
3. Cancel order
4. Confirm cancellation

The harness should capture these actions.

A simple model can represent a tool invocation:

public sealed record ToolCall(
    string Name,
    IReadOnlyDictionary<string, object?> Arguments);

Then verify the expected behavior:

[Fact]
public void CancellationUsesExpectedTool()
{
    var call = new ToolCall(
        "CancelOrder",
        new Dictionary<string, object?>
        {
            ["orderId"] = "ORD-1001"
        });

    Assert.Equal(
        "CancelOrder",
        call.Name);

    Assert.Equal(
        "ORD-1001",
        call.Arguments["orderId"]);
}

This catches errors that may not be visible in the final answer.

Verify Tool Arguments

A correct tool with incorrect arguments can still cause a serious problem.

For example:

Tool: CancelOrder

Expected:
orderId = ORD-1001

Actual:
orderId = ORD-1002

The agent might produce a perfectly reasonable confirmation message while performing the wrong operation.

For state-changing tools, argument validation should therefore be stricter than for informational tools.

Classify Tool Risk

A useful approach is to classify tools by impact.

Read
  ↓
Low Risk

Write
  ↓
Medium Risk

Financial / Destructive
  ↓
High Risk

For example:

Tool	Risk	Verification
SearchProducts	Low	Input validation
GetCustomer	Low	Authorization
UpdateProfile	Medium	Authorization + validation
CancelOrder	High	Authorization + confirmation
IssueRefund	Critical	Strong authorization + business rules

The exact classification depends on the application.

AI-Based Evaluation

Deterministic checks cannot answer every question.

Consider:

"Was the response clear and relevant?"

There may not be a single exact expected string.

An AI evaluator can assess characteristics such as:

Relevance
Completeness
Clarity
Grounding
Instruction adherence
Response quality

A structured evaluation result can be represented as:

public sealed record AiEvaluation(
    double Score,
    bool Passed,
    string Reason);

For example:

{
  "score": 4,
  "passed": true,
  "reason": "The response directly answered the request and used the provided customer information."
}

The evaluator should use a defined rubric rather than an unrestricted question such as:

"Is this good?"

Designing an Evaluation Rubric

A rubric makes AI-based evaluation more consistent.

For example:

Score 1:
Incorrect or irrelevant

Score 2:
Partially useful but major omissions

Score 3:
Acceptable response

Score 4:
Correct and complete

Score 5:
Correct, complete, clear, and appropriately grounded

A structured evaluator prompt can then request:

Evaluate the response using the supplied rubric.

Check:
- Factual correctness
- Relevance
- Completeness
- Grounding
- Instruction adherence

Return:
- Score
- Pass/fail
- Short reason

The evaluator's own variability should still be considered.

Do Not Use AI Evaluation for Security Authorization

An AI evaluator should not determine whether an operation is authorized.

For example, avoid:

AI:
"Should this user be allowed to issue a refund?"

Instead:

Authorization Service
        ↓
Allowed / Denied
        ↓
Agent

Security and business authorization should remain deterministic.

AI evaluation can assess whether the agent's response correctly communicated the result, but not whether the user actually has permission.

Human Review

Some scenarios are too important or ambiguous for automated evaluation alone.

Human review is appropriate for:

High-impact decisions
Security-sensitive workflows
New agent behaviors
Evaluation disagreements
Low-confidence results
Unexpected tool sequences

A review queue can be represented as:

Agent Execution
      ↓
Automated Verification
      |
      +---- Clearly Passed → Accept
      |
      +---- Clearly Failed → Reject
      |
      +---- Uncertain → Human Review

This reduces the amount of manual review while preserving human oversight where it matters.

Confidence-Based Routing

Suppose an evaluator produces:

Score: 4.8
Confidence: High

The case may be automatically accepted.

If it produces:

Score: 2.7
Confidence: Low

send it to review.

A model can represent this:

public sealed record VerificationDecision(
    string Status,
    double Confidence);

Then:

public static string Route(
    VerificationDecision decision)
{
    if (decision.Confidence < 0.70)
    {
        return "HumanReview";
    }

    return decision.Status;
}

The threshold should be determined through validation rather than chosen arbitrarily for a production system.

Evaluation Datasets

Create a version-controlled collection of representative scenarios.

[
  {
    "id": "order-001",
    "input": "Where is my order?",
    "expectedBehavior": "Return current order status."
  },
  {
    "id": "order-002",
    "input": "Cancel my order.",
    "expectedBehavior": "Verify authorization before cancellation."
  }
]

Each evaluation case can include:

Input
Expected behavior
Risk level
Required tools
Forbidden actions
Expected business rules
Evaluation criteria

This transforms testing from an ad hoc process into a repeatable verification suite.

Testing Non-Deterministic Responses

Do not always compare generated text exactly.

This is fragile:

Assert.Equal(
    "Your order has been cancelled.",
    response);

A semantically equivalent response could be:

"Order ORD-1001 has been successfully cancelled."

Instead, test deterministic properties:

Assert.Contains(
    "cancelled",
    response,
    StringComparison.OrdinalIgnoreCase);

For more complex responses, use structured output or an evaluator.

The verification target should be the behavior that matters, not unnecessary wording details.

Testing Grounded Responses

If an agent uses retrieved information, verify that the response is grounded in the available data.

For example:

Retrieved Order:
Status = Shipped

Agent Response:
"Your order is currently shipped."

But:

Retrieved Order:
Status = Shipped

Agent Response:
"Your order will arrive tomorrow at 2 PM."

The second statement introduces information that was not present in the supplied data.

A grounding evaluator can flag unsupported claims.

Verification Pipeline

A complete evaluation can follow:

Test Case
   ↓
Agent Execution
   ↓
Capture Trace
   ↓
Validate Tool Calls
   ↓
Validate Business Rules
   ↓
Validate Security Rules
   ↓
Evaluate Response
   ↓
Determine Confidence
   ↓
Human Review if Required
   ↓
Final Result

This creates multiple opportunities to catch failures.

Example Verification Orchestrator

A simplified C# orchestrator could look like:

public sealed class AgentVerifier
{
    public async Task<VerificationResult> VerifyAsync(
        EvaluationCase testCase,
        CancellationToken cancellationToken)
    {
        var execution =
            await RunAgentAsync(
                testCase,
                cancellationToken);

        var deterministicResult =
            ValidateDeterministicRules(execution);

        if (!deterministicResult.Passed)
        {
            return VerificationResult.Failed(
                deterministicResult.Reason);
        }

        var aiResult =
            await EvaluateResponseAsync(
                execution,
                cancellationToken);

        if (aiResult.Confidence < 0.70)
        {
            return VerificationResult.NeedsReview(
                "Evaluation confidence is low.");
        }

        return aiResult.Passed
            ? VerificationResult.Passed()
            : VerificationResult.Failed(
                aiResult.Reason);
    }
}

The example intentionally keeps the implementation simple. In a production system, the verification stages should be independently testable and observable.

Observability and Verification

Verification becomes significantly more useful when execution traces are available.

A trace might show:

Agent Request
   |
   +---- Model Call
   |
   +---- GetCustomer
   |
   +---- GetOrder
   |
   +---- CancelOrder
   |
   +---- Model Call
   |
   +---- Final Response

If the final result fails evaluation, developers can inspect the execution path rather than guessing what happened.

OpenTelemetry can be used to instrument application-level operations and propagate trace context through supported dependencies.

Verification Results

Store structured results instead of only writing pass/fail messages to a log.

public sealed record VerificationResult(
    string TestId,
    bool Passed,
    string Status,
    string? Reason,
    TimeSpan Duration);

Possible statuses include:

Passed
Failed
NeedsReview
Blocked
Error

This provides a useful distinction between an actual agent failure and an infrastructure problem.

Common Mistakes

Testing Only the Final Response

The agent may have taken an unsafe path while producing a convincing answer.

Allowing AI to Enforce Authorization

Authorization should remain deterministic.

Comparing Exact Text

Natural-language responses can vary while remaining correct.

Ignoring Tool Arguments

Calling the correct tool with the wrong identifier can still cause a serious failure.

Sending Every Case to Humans

Manual review does not scale.

Automate deterministic checks and route only uncertain or high-risk cases to people.

Using Only One Evaluator

For critical systems, combine multiple verification mechanisms instead of relying entirely on an AI judge.

Best Practices

Separate deterministic rules from AI-based evaluation.
Keep authorization and security controls deterministic.
Validate tool selection and arguments.
Use structured evaluation datasets.
Avoid exact text matching for open-ended responses.
Classify tools by risk.
Route uncertain cases to human reviewers.
Capture execution traces.
Store structured verification results.
Version evaluation datasets and rubrics.
Test both successful and adversarial scenarios.
Monitor evaluation failures over time.
Re-run regression evaluations after model or prompt changes.
Keep sensitive data out of unnecessary traces and evaluation records.

Advantages and Disadvantages

Advantages

Combines automated and human verification
Protects deterministic business rules
Detects incorrect tool usage
Supports open-ended response evaluation
Scales better than manual review alone
Provides structured regression testing
Improves visibility into agent failures

Disadvantages

More complex than traditional unit testing
AI evaluators can introduce variability
Human review is still required for some cases
Evaluation datasets require maintenance
Trace data can contain sensitive information
Multiple verification layers increase implementation effort

Troubleshooting Verification Failures

When an evaluation fails, inspect the layers in order:

Verify the test input.
Check the agent execution.
Inspect tool calls.
Validate tool arguments.
Check deterministic business rules.
Review authorization decisions.
Inspect retrieved data.
Evaluate the final response.
Review evaluator confidence.
Determine whether human review is necessary.

This avoids changing the AI prompt when the actual problem may be an application bug or incorrect test expectation.

A Practical Enterprise Architecture

A production-oriented verification platform can be organized as:

                  Agent Application
                         |
                         v
                  Execution Harness
                         |
          +--------------+--------------+
          |              |              |
          v              v              v
     Trace Capture   Rule Engine    AI Evaluator
          |              |              |
          +--------------+--------------+
                         |
                         v
                  Decision Engine
                         |
              +----------+----------+
              |                     |
              v                     v
          Auto Accept          Human Review

The decision engine can combine:

Deterministic test results
Security results
Tool validation
AI evaluation
Confidence
Risk classification

This creates a scalable verification model without pretending that every AI decision can be reduced to a single deterministic assertion.

Conclusion

AI agents require a different verification strategy from traditional deterministic software. The goal is not to force every model response to be identical. Instead, teams should define the behaviors and invariants that must remain reliable regardless of how the agent reaches its conclusion.

A strong verification framework combines deterministic tests, AI-based evaluation, execution tracing, risk-aware tool validation, and human review.

Deterministic code should enforce authorization, business rules, and other critical constraints. AI evaluators can assess open-ended qualities such as relevance and completeness. Human reviewers should handle ambiguous or high-impact cases.

The resulting architecture creates a practical balance between automation and control:

Deterministic Rules
        +
AI Evaluation
        +
Human Oversight
        ↓
Reliable Agent Verification

This approach allows organizations to adopt agentic AI while maintaining the engineering discipline required for systems that operate on real data, invoke real tools, and participate in business-critical workflows.