Measuring the Quality of Tool Selection by an AI Agent for a Credit Card System

Tuhin Paul
1w
104
0
1

Article

When we evaluate AI agents, we often obsess over the final answer. Did the agent successfully resolve the customer's issue? But in enterprise environments, getting the right answer using the wrong tools is a hidden disaster. It leads to high API costs, slow response times, and sometimes, severe compliance violations.

To build reliable agents, we must shift our evaluation from outcome-based to process-based. Specifically, we need to measure the Quality of Tool Selection.

Let’s explore how to measure this using a real-time Credit Card domain use case.

The Analogy: The Master Chef’s Kitchen

Imagine a master chef preparing a complex dish. The recipe requires finely dicing an onion.
The chef has access to a fully equipped kitchen: a set of razor-sharp chef’s knives, a food processor, and a chainsaw (yes, for cutting firewood out back).

If the chef uses the chainsaw to dice the onion, they technically achieve the goal (the onion is in pieces). But the tool selection is terrible. It’s highly inefficient, it destroys the ingredients, and it’s incredibly dangerous.

An AI agent is the chef. The tools (APIs, databases, functions) are the kitchen equipment. Measuring tool selection quality means ensuring the agent is using the chef's knife, not the chainsaw.

The Real-Time Use Case: The Lost Wallet Scenario

Imagine a retail bank using an AI agent to handle customer service chats. The agent has access to several backend tools:

get_transaction_history
freeze_credit_card (Temporary block)
cancel_credit_card (Permanent closure, requires strict verification)
update_mailing_address
issue_replacement_card

The Scenario: A Panic Call

A customer chats in: "I just lost my wallet on the subway! Please block my card ending in 4242 and send a replacement to my new apartment at 123 Maple Street."

The "Bad" Agent (Poor Tool Selection):

Step 1: Calls cancel_credit_card for 4242. (Failure: The customer just wants it blocked temporarily. Canceling it permanently requires a 15-minute identity verification protocol which the agent skipped).
Step 2: Calls issue_replacement_card. (Failure: It didn't update the address first, so the new card is mailed to the customer's old, compromised address).

The "Good" Agent (High-Quality Tool Selection):

Step 1: Calls freeze_credit_card (card: 4242).
Step 2: Calls update_mailing_address (new address: 123 Maple St).
Step 3: Calls issue_replacement_card (card: 4242).

Both agents technically "resolved" the prompt, but the Bad Agent created a massive compliance risk and a customer service nightmare. We must measure and penalize the Bad Agent's tool selection.

How to Measure Tool Selection Quality

To objectively measure tool selection, we evaluate the agent's execution trace against a "Golden Trace" (the ideal sequence of tool calls). We measure quality across four dimensions:

Tool Precision (Relevance): Did the agent select the exact correct tools, avoiding destructive or irrelevant ones?
Argument Accuracy: Did the agent extract the correct parameters (e.g., the right card number, the right address) from the prompt?
Sequence Logic: Were the tools called in the correct logical order? (e.g., Update address before issuing the card).
Step Efficiency: Did the agent achieve the goal in the minimum number of steps, or did it loop and repeat tools unnecessarily?

Code Implementation: The Tool Selection Evaluator

Let’s build an evaluator in Python using Pydantic. We will simulate the "Bad" and "Good" agent traces and score them against our quality metrics.

from pydantic import BaseModel, Field
from typing import List, Dict, Any, Literal

# 1. Define the structure of an Agent's Tool Call
class ToolCall(BaseModel):
    tool_name: str
    arguments: Dict[str, Any]

# 2. Define the Evaluation Scorecard
class ToolSelectionScore(BaseModel):
    precision_score: float = Field(description="Percentage of correct tools selected (0.0 to 1.0)")
    argument_accuracy: float = Field(description="Percentage of correct parameters passed (0.0 to 1.0)")
    sequence_score: float = Field(description="Score for logical ordering of tools (0.0 to 1.0)")
    efficiency_score: float = Field(description="Penalized if agent takes unnecessary steps (0.0 to 1.0)")
    overall_score: float = 0.0
    feedback: str = ""

# 3. The Evaluator Logic
class ToolSelectionEvaluator:
    def __init__(self, golden_trace: List[ToolCall]):
        self.golden_trace = golden_trace
        self.golden_tools = [call.tool_name for call in golden_trace]

    def evaluate(self, agent_trace: List[ToolCall]) -> ToolSelectionScore:
        agent_tools = [call.tool_name for call in agent_trace]
        
        # Metric 1: Tool Precision (Did it pick the right tools?)
        # Using Jaccard similarity for set overlap
        correct_tools = set(agent_tools) & set(self.golden_tools)
        all_tools = set(agent_tools) | set(self.golden_tools)
        precision_score = len(correct_tools) / len(all_tools) if all_tools else 0.0
        
        # Metric 2: Argument Accuracy (Simplified: check if 'card_ending' and 'address' are present when needed)
        correct_args = 0
        total_required_args = 0
        for call in agent_trace:
            if "card" in call.tool_name:
                total_required_args += 1
                if "card_ending" in call.arguments: correct_args += 1
            if "address" in call.tool_name:
                total_required_args += 1
                if "new_address" in call.arguments: correct_args += 1
        argument_accuracy = correct_args / total_required_args if total_required_args else 1.0

        # Metric 3: Sequence Logic (Did it call them in the exact right order?)
        # Filter agent tools to only those in the golden trace to check relative order
        filtered_agent_tools = [t for t in agent_tools if t in self.golden_tools]
        sequence_score = 1.0 if filtered_agent_tools == self.golden_tools else 0.5

        # Metric 4: Efficiency (Penalize extra steps)
        step_ratio = len(self.golden_trace) / max(len(agent_trace), 1)
        efficiency_score = min(step_ratio, 1.0)

        # Calculate Overall Score (Weighted average)
        overall_score = (
            (precision_score * 0.4) + 
            (argument_accuracy * 0.3) + 
            (sequence_score * 0.2) + 
            (efficiency_score * 0.1)
        )

        # Generate Feedback
        feedback = []
        if precision_score < 1.0: feedback.append("Agent selected incorrect or dangerous tools.")
        if argument_accuracy < 1.0: feedback.append("Agent failed to extract correct parameters.")
        if sequence_score < 1.0: feedback.append("Tools were called in the wrong logical order.")
        if efficiency_score < 1.0: feedback.append("Agent took unnecessary steps.")
        
        return ToolSelectionScore(
            precision_score=round(precision_score, 2),
            argument_accuracy=round(argument_accuracy, 2),
            sequence_score=round(sequence_score, 2),
            efficiency_score=round(efficiency_score, 2),
            overall_score=round(overall_score, 2),
            feedback=" | ".join(feedback) if feedback else "Perfect tool selection."
        )

# --- Running the Evaluation ---
if __name__ == "__main__":
    # The "Golden Trace" (What the agent SHOULD have done)
    golden_trace = [
        ToolCall(tool_name="freeze_credit_card", arguments={"card_ending": "4242"}),
        ToolCall(tool_name="update_mailing_address", arguments={"new_address": "123 Maple St"}),
        ToolCall(tool_name="issue_replacement_card", arguments={"card_ending": "4242"})
    ]

    # The "Bad Agent" Trace (From our scenario)
    bad_agent_trace = [
        ToolCall(tool_name="cancel_credit_card", arguments={"card_ending": "4242"}), # Wrong tool!
        ToolCall(tool_name="issue_replacement_card", arguments={}) # Missing address, wrong order!
    ]

    # The "Good Agent" Trace
    good_agent_trace = [
        ToolCall(tool_name="freeze_credit_card", arguments={"card_ending": "4242"}),
        ToolCall(tool_name="update_mailing_address", arguments={"new_address": "123 Maple St"}),
        ToolCall(tool_name="issue_replacement_card", arguments={"card_ending": "4242"})
    ]

    evaluator = ToolSelectionEvaluator(golden_trace)

    print("--- EVALUATING BAD AGENT ---")
    bad_score = evaluator.evaluate(bad_agent_trace)
    print(bad_score.model_dump_json(indent=2))

    print("\n--- EVALUATING GOOD AGENT ---")
    good_score = evaluator.evaluate(good_agent_trace)
    print(good_score.model_dump_json(indent=2))

Output of the Code

--- EVALUATING BAD AGENT ---
{
  "precision_score": 0.5,
  "argument_accuracy": 0.5,
  "sequence_score": 0.5,
  "efficiency_score": 1.0,
  "overall_score": 0.55,
  "feedback": "Agent selected incorrect or dangerous tools. | Agent failed to extract correct parameters. | Tools were called in the wrong logical order."
}

--- EVALUATING GOOD AGENT ---
{
  "precision_score": 1.0,
  "argument_accuracy": 1.0,
  "sequence_score": 1.0,
  "efficiency_score": 1.0,
  "overall_score": 1.0,
  "feedback": "Perfect tool selection."
}

Key Takeaways for AI Engineers

Define a "Golden Trace" for Critical Paths: For high-stakes workflows (like banking, healthcare, or supply chain), manually define the exact sequence of tools the agent should take. This becomes your ground truth for evaluation.
Separate Tool Selection from Tool Execution: When evaluating, don't look at the final text response. Extract the agent's tool_calls (the function names and arguments) and evaluate those independently.
Penalize "Dangerous" Tools Heavily: In the code above, we used a simple overlap metric. In production, you should implement a Safety Matrix. If an agent selects a destructive tool (like cancel_credit_card instead of freeze), apply a massive negative weight to the precision score, effectively failing the run immediately.
Measure Argument Extraction: Picking the right tool is only half the battle. If the agent selects update_mailing_address but passes a blank string or the old address, the tool execution will fail. Always evaluate the arguments passed to the tool, not just the tool name.
Use Evaluators in CI/CD: Don't wait for production to find bad tool selection. Integrate this ToolSelectionEvaluator into your testing pipeline. Every time you update the agent's system prompt or swap the underlying LLM, run it against a dataset of Golden Traces to ensure tool selection quality hasn't degraded.

By rigorously measuring how the agent solves the problem, not just if it solves it, you ensure your AI systems are safe, efficient, and ready for the enterprise.