As AI agents become connected to more enterprise systems, the number of tools available to a model can grow very quickly.
A small agent might use five tools:
search_customer
get_order
create_ticket
update_ticket
send_notification
An enterprise agent can easily have hundreds of tools across systems such as GitHub, monitoring platforms, ticketing systems, databases, analytics services, and internal APIs.
That creates a problem that is easy to miss during early development: giving an AI agent more tools does not necessarily make it more capable.
A large tool catalog consumes context, increases input-token usage, and can make tool selection more difficult. Microsoft Foundry's current tool-search capability addresses this by hiding the full tool catalog and exposing discovery mechanisms that retrieve relevant tools when they are needed.
Claude models hosted in Microsoft Foundry now support Tool Search alongside capabilities such as structured outputs, web search, web fetch, and MCP connectivity.
This makes an interesting benchmark possible:
How does Claude perform when every tool definition is loaded up front compared with discovering tools dynamically?
The answer should be evaluated using measurable metrics such as tool-selection accuracy, token usage, latency, failed tool calls, and task completion.
Static Tools vs Tool Search
There are two fundamentally different approaches.
Static Tool Loading
In the traditional approach, the agent receives the available tool definitions at the beginning of the request.
User Request
|
v
Claude
|
+-- Tool A
+-- Tool B
+-- Tool C
+-- Tool D
+-- Tool E
+-- Tool F
+-- Tool G
+-- ...
Every tool definition becomes part of the model's available context.
This works well when the tool catalog is small.
Dynamic Tool Search
Tool Search changes the loading model.
Instead of exposing every tool immediately, the agent can search for the capability it needs.
User Request
|
v
Claude
|
v
tool_search
|
v
Relevant Tools
|
v
call_tool
Microsoft Foundry's toolbox implementation uses tool_search to discover relevant tools and call_tool to invoke a discovered tool. Tool Search uses tool names, descriptions, and parameter information when determining which tools match the requested capability.
Why Tool Count Matters
Imagine an enterprise agent with 500 tools.
With static loading:
500 tool definitions
|
v
Large context
|
+-- Conversation history
|
+-- System instructions
|
+-- User request
|
+-- Retrieved data
The model has to reason in an environment containing a large amount of tool metadata.
The problem becomes more pronounced as tools have detailed schemas.
Microsoft's documentation specifically identifies three problems with large toolboxes:
Increased token consumption
Reduced available context
Less accurate tool selection
Tool Search attempts to address all three.
How Microsoft Foundry Tool Search Works
When Tool Search is enabled for a toolbox, the initial tool list does not expose all toolbox tools.
Instead, the model receives the discovery interface.
Conceptually:
Initial Context
|
+-- tool_search
|
+-- call_tool
|
+-- Conversation
When the model determines that it needs a particular capability, it performs a search.
For example:
Query:
"Find the customer's recent support incidents"
The system searches the tool catalog and returns the most relevant tools.
The model can then invoke the discovered tool.
Microsoft documents BM25 as the ranking mechanism for toolbox Tool Search. The search considers tool names, descriptions, and parameter information.
The Benchmark Question
A useful benchmark should compare:
Configuration A:
All tools statically available
versus
Configuration B:
Tool Search enabled
The model, task, tool catalog, and environment should remain the same.
Only the tool-discovery strategy should change.
Benchmark Environment
Start with a representative toolbox.
For example:
GitHub
├── Search repositories
├── Search issues
├── Search pull requests
├── Get pull request
└── Create issue
Monitoring
├── Search incidents
├── Get incident
├── Search alerts
└── Get service health
Customer System
├── Search customers
├── Get customer
├── Search orders
└── Get order
Communication
├── Search messages
├── Send message
└── Create notification
The benchmark can then scale this catalog:
10 tools
25 tools
50 tools
100 tools
250 tools
500 tools
1000 tools
This is important because Tool Search is primarily interesting when the tool catalog becomes large.
Microsoft currently recommends Tool Search for larger toolboxes, particularly when there are more than roughly 10–15 tools or when different tasks need different subsets of the catalog.
Benchmark Tasks
The benchmark should contain realistic agent tasks rather than artificial tool-selection questions.
Task 1: Customer Support
Find the customer's latest unresolved incident
and summarize its current status.
Expected tools:
search_customer
search_incidents
get_incident
Task 2: Engineering
Find the open pull request related to the
authentication timeout issue.
Expected tools:
search_pull_requests
get_pull_request
Task 3: Operations
Identify the active checkout latency incident
and retrieve its current service health.
Expected tools:
search_incidents
get_service_health
Task 4: Cross-System Workflow
Find the customer's latest failed order,
check whether an incident exists, and create
a support ticket if necessary.
Expected tools:
search_customer
search_orders
search_incidents
create_ticket
These tasks test both discovery and execution.
Measuring Tool Selection Accuracy
The first important metric is whether the agent selects the correct tool.
Define:
Tool Selection Accuracy =
Correct Tool Selections / Total Tool Selections
For example:
100 tool selections
92 correct
8 incorrect
Accuracy = 92%
But accuracy alone is not enough.
An agent may eventually find the correct tool after several incorrect searches.
Therefore, also measure:
First-Choice Accuracy
and:
Recovery Accuracy
These provide more detail.
First-Choice Accuracy
Suppose the correct tool is:
search_pull_requests
The agent chooses:
search_issues
first and then discovers the correct tool.
The final task may succeed, but the first-choice selection was incorrect.
This distinction matters in large tool catalogs.
First choice
|
X Wrong
|
v
Second search
|
v
Correct tool
Dynamic discovery should ideally reduce these errors by narrowing the candidate set.
Measuring Token Consumption
Tool definitions consume input context.
A static configuration might repeatedly send:
Tool A schema
Tool B schema
Tool C schema
...
Tool N schema
A dynamic configuration can instead keep only the relevant working set.
Microsoft states that Tool Search can keep the active tool context focused and reports that its Claude implementation can reduce tool-definition tokens substantially in large catalogs. These published figures are platform claims rather than an independent benchmark, so they should be validated against your own workload.
A benchmark should measure actual input tokens.
Static:
Input Tokens = Prompt + All Tool Definitions
Tool Search:
Input Tokens = Prompt + Search Interface + Retrieved Tools
Then calculate:
Token Reduction % =
(Static Tokens - Dynamic Tokens)
/
Static Tokens × 100
Measuring Latency
Tool Search introduces an additional discovery operation.
That creates a trade-off.
Static:
Request
|
v
Model
|
v
Tool
versus:
Tool Search:
Request
|
v
Model
|
v
Search
|
v
Tool
Dynamic discovery can reduce context size while adding search overhead.
Therefore, measure:
Initial model latency
Tool-search latency
Tool execution latency
Total task latency
A useful result table is:
| Metric | Static | Tool Search |
|---|
| Median latency | Measure | Measure |
| p95 latency | Measure | Measure |
| Tool-search overhead | N/A | Measure |
| Input tokens | Measure | Measure |
| Failed selections | Measure | Measure |
Do not assume Tool Search will always be faster.
The benchmark needs to determine the trade-off.
Measuring Tool-Selection Errors
A large tool catalog may contain tools with similar names.
For example:
search_issue
search_issues
search_pull_request
search_pull_requests
search_incident
search_incidents
The descriptions become extremely important.
Test cases should intentionally include similar tool names.
This helps determine whether the search mechanism can distinguish them reliably.
Tool Description Quality
Tool Search depends heavily on metadata quality.
Microsoft's documentation notes that tool names and descriptions influence discovery and recommends giving every tool a clear description. Vague or missing descriptions can produce poor search results.
Compare:
Name:
search_data
Description:
Search data.
with:
Name:
search_customer_orders
Description:
Find customer orders by customer ID,
order ID, status, or date range.
Use this when investigating customer
purchase history.
The second description provides substantially more semantic information.
Benchmarking Description Quality
Create two versions of the same toolbox.
Version A
Poor descriptions:
get_data
search_data
update_data
Version B
Intent-rich descriptions:
search_customer_orders
get_customer_order
update_order_status
Then compare Tool Search accuracy.
This separates two questions:
Does Tool Search work?
from:
Does Tool Search work with our tool metadata?
The second question is often more useful in production.
Pinning Critical Tools
Tool Search does not mean every tool must always be discovered dynamically.
Microsoft Foundry supports pinning important tools so they remain directly visible without a search round-trip.
Conceptually:
Always Available
├── get_current_user
└── submit_response
Discovered Dynamically
├── GitHub tools
├── Monitoring tools
├── Database tools
└── Customer tools
This hybrid strategy can be valuable when a small set of tools is used on almost every request.
Benchmarking Pinned vs Unpinned Tools
Add another experiment:
A:
Everything static
B:
Everything searchable
C:
Critical tools pinned
Everything else searchable
Then compare:
| Configuration | Token Cost | Latency | Selection Accuracy |
|---|
| Static | Measure | Measure | Measure |
| Search | Measure | Measure | Measure |
| Hybrid | Measure | Measure | Measure |
This is often more representative of a production architecture than a simple static-vs-search comparison.
Scaling the Tool Catalog
The most important experiment is to increase tool count.
For example:
10
25
50
100
250
500
1000
At each level, run the same tasks.
Then plot:
Tool Count
X
|
v
Token Usage
Selection Accuracy
Latency
The objective is to identify the point at which static loading becomes operationally expensive or unreliable for your workload.
Example Benchmark Dataset
A benchmark record could look like:
{
"configuration": "tool-search",
"toolCount": 250,
"task": "support-incident",
"firstChoiceCorrect": true,
"finalToolCorrect": true,
"toolSearches": 1,
"toolCalls": 3,
"inputTokens": 8200,
"durationMs": 1840,
"success": true
}
For static tools:
{
"configuration": "static",
"toolCount": 250,
"task": "support-incident",
"firstChoiceCorrect": false,
"finalToolCorrect": true,
"toolSearches": 0,
"toolCalls": 4,
"inputTokens": 38200,
"durationMs": 2190,
"success": true
}
These values are illustrative, not measured results.
The benchmark should populate the fields using actual traces.
Testing Large Enterprise Catalogs
Enterprise tool catalogs can be particularly challenging because different systems often use different naming conventions.
For example:
GitHub:
search_pull_requests
Monitoring:
query_incident_records
CRM:
find_customer_cases
Internal:
lookup_support_requests
A user may ask:
Find the customer's open support issue.
The tool names may not contain the exact phrase "support issue."
This is where additional_search_text can become useful.
Microsoft Foundry allows additional search text to be added to a tool's search index without changing the tool schema shown to the model.
For example:
Tool:
lookup_support_requests
Additional search terms:
customer issue, support ticket,
case, incident, complaint
This can help bridge differences between organizational vocabulary and tool names.
Failure Testing
The benchmark should also test when Tool Search returns the wrong tool.
For example:
User:
Find the open pull request for issue 4821.
If the agent receives:
search_issues
instead of:
search_pull_requests
measure what happens next.
A resilient agent should recognize the mismatch rather than repeatedly calling the wrong tool.
Test:
Correct first search
Wrong first search
No result
Multiple similar results
Ambiguous request
Benchmarking Ambiguous Requests
Real users rarely formulate perfect tool-search queries.
For example:
"Check the checkout problem."
could mean:
Search incidents
Search alerts
Check service health
Search customer tickets
A good benchmark should include ambiguous requests.
This tests whether the agent can use the discovered tool information and conversation context appropriately.
Common Mistakes
Assuming Fewer Tools Automatically Means Better Performance
Tool Search can reduce context, but discovery itself introduces work.
Measure the complete workflow.
Ignoring Tool Descriptions
Poor metadata can make a sophisticated search mechanism look ineffective.
Using Only Synthetic Tool Names
Real enterprise catalogs contain inconsistent naming conventions.
Use realistic metadata.
Measuring Only Token Savings
Lower token usage does not automatically mean higher task accuracy.
Ignoring Search Latency
A search round-trip has a cost.
Measure it.
Making Every Tool Dynamic
Some tools may be needed on nearly every request.
Pinning can avoid unnecessary discovery.
Troubleshooting Poor Tool Discovery
If Tool Search frequently returns the wrong tools:
Improve tool names.
Add precise descriptions.
Describe the intended use case.
Include important parameter terminology.
Add organization-specific search terms.
Pin frequently used critical tools.
Test searches against similar tool names.
Inspect the toolbox version being executed.
Microsoft recommends versioning toolboxes and testing a version-specific endpoint before promoting it as the default.
This is especially useful when changing tool descriptions or discovery configuration.
Best Practices
Keep Tool Descriptions Intent-Oriented
Describe both what the tool does and when it should be used.
Use Tool Search for Large Catalogs
The value increases as the tool catalog grows and different tasks use different subsets.
Pin High-Frequency Tools
Avoid a discovery round-trip for capabilities required on nearly every request.
Measure p50 and p95
Average latency can hide occasional expensive discovery or execution paths.
Track Tool-Selection Accuracy
Do not evaluate the feature only through token counts.
Version Your Toolbox
Treat tool definitions and discovery configuration as deployable artifacts.
Build Evaluation Tasks Around Real User Requests
Production queries are often ambiguous, incomplete, or expressed using internal terminology.
Advantages and Disadvantages
Advantages
Reduces the amount of tool metadata loaded into the active context.
Can improve tool selection in large catalogs.
Helps control input-token usage.
Allows tools to be discovered based on intent.
Supports pinning for frequently used capabilities.
Fits naturally with large enterprise toolboxes.
Disadvantages
Adds a discovery step.
Search quality depends heavily on tool metadata.
Ambiguous requests can still produce poor matches.
Requires additional evaluation and observability.
Tool availability and behavior may vary by Foundry feature version and deployment configuration.
Final Thoughts
The most interesting part of Tool Search is not simply that it can reduce the number of tool definitions sent to a model. It changes the architecture of tool-enabled agents from load everything first to discover capabilities when needed.
Microsoft Foundry's implementation uses tool_search and call_tool to keep the active tool set focused, while allowing the complete toolbox to remain available for discovery.
For Claude-based agents, this creates a measurable engineering trade-off. Static tools can be extremely effective when the catalog is small and predictable. Tool Search becomes more attractive as the catalog grows, particularly when different requests require different subsets of capabilities.
The right decision should therefore come from benchmarking your own workload.
Measure tool-selection accuracy, first-choice accuracy, input tokens, search latency, total latency, failed tool calls, and task success as the catalog scales from a handful of tools to hundreds or thousands.
That benchmark will tell you much more than a simple claim that dynamic tool discovery is "better." It will show exactly when it becomes better, why it becomes better, and where static tools may still be the more efficient choice.