Claude in Microsoft Foundry: Benchmarking Tool Search vs Static Tools

As AI agents become connected to more enterprise systems, the number of tools available to a model can grow very quickly.

A small agent might use five tools:

search_customer
get_order
create_ticket
update_ticket
send_notification

An enterprise agent can easily have hundreds of tools across systems such as GitHub, monitoring platforms, ticketing systems, databases, analytics services, and internal APIs.

That creates a problem that is easy to miss during early development: giving an AI agent more tools does not necessarily make it more capable.

A large tool catalog consumes context, increases input-token usage, and can make tool selection more difficult. Microsoft Foundry's current tool-search capability addresses this by hiding the full tool catalog and exposing discovery mechanisms that retrieve relevant tools when they are needed.

Claude models hosted in Microsoft Foundry now support Tool Search alongside capabilities such as structured outputs, web search, web fetch, and MCP connectivity.

This makes an interesting benchmark possible:

How does Claude perform when every tool definition is loaded up front compared with discovering tools dynamically?

The answer should be evaluated using measurable metrics such as tool-selection accuracy, token usage, latency, failed tool calls, and task completion.

Static Tools vs Tool Search

There are two fundamentally different approaches.

Static Tool Loading

In the traditional approach, the agent receives the available tool definitions at the beginning of the request.

User Request
     |
     v
Claude
     |
     +-- Tool A
     +-- Tool B
     +-- Tool C
     +-- Tool D
     +-- Tool E
     +-- Tool F
     +-- Tool G
     +-- ...

Every tool definition becomes part of the model's available context.

This works well when the tool catalog is small.

Dynamic Tool Search

Tool Search changes the loading model.

Instead of exposing every tool immediately, the agent can search for the capability it needs.

User Request
     |
     v
Claude
     |
     v
tool_search
     |
     v
Relevant Tools
     |
     v
call_tool

Microsoft Foundry's toolbox implementation uses tool_search to discover relevant tools and call_tool to invoke a discovered tool. Tool Search uses tool names, descriptions, and parameter information when determining which tools match the requested capability.

Why Tool Count Matters

Imagine an enterprise agent with 500 tools.

With static loading:

500 tool definitions
        |
        v
Large context
        |
        +-- Conversation history
        |
        +-- System instructions
        |
        +-- User request
        |
        +-- Retrieved data

The model has to reason in an environment containing a large amount of tool metadata.

The problem becomes more pronounced as tools have detailed schemas.

Microsoft's documentation specifically identifies three problems with large toolboxes:

Increased token consumption
Reduced available context
Less accurate tool selection

Tool Search attempts to address all three.

How Microsoft Foundry Tool Search Works

When Tool Search is enabled for a toolbox, the initial tool list does not expose all toolbox tools.

Instead, the model receives the discovery interface.

Conceptually:

Initial Context
      |
      +-- tool_search
      |
      +-- call_tool
      |
      +-- Conversation

When the model determines that it needs a particular capability, it performs a search.

For example:

Query:
"Find the customer's recent support incidents"

The system searches the tool catalog and returns the most relevant tools.

The model can then invoke the discovered tool.

Microsoft documents BM25 as the ranking mechanism for toolbox Tool Search. The search considers tool names, descriptions, and parameter information.

The Benchmark Question

A useful benchmark should compare:

Configuration A:
All tools statically available

versus

Configuration B:
Tool Search enabled

The model, task, tool catalog, and environment should remain the same.

Only the tool-discovery strategy should change.

Benchmark Environment

Start with a representative toolbox.

For example:

GitHub
 ├── Search repositories
 ├── Search issues
 ├── Search pull requests
 ├── Get pull request
 └── Create issue

Monitoring
 ├── Search incidents
 ├── Get incident
 ├── Search alerts
 └── Get service health

Customer System
 ├── Search customers
 ├── Get customer
 ├── Search orders
 └── Get order

Communication
 ├── Search messages
 ├── Send message
 └── Create notification

The benchmark can then scale this catalog:

10 tools
25 tools
50 tools
100 tools
250 tools
500 tools
1000 tools

This is important because Tool Search is primarily interesting when the tool catalog becomes large.

Microsoft currently recommends Tool Search for larger toolboxes, particularly when there are more than roughly 10–15 tools or when different tasks need different subsets of the catalog.

Benchmark Tasks

The benchmark should contain realistic agent tasks rather than artificial tool-selection questions.

Task 1: Customer Support

Find the customer's latest unresolved incident
and summarize its current status.

Expected tools:

search_customer
search_incidents
get_incident

Task 2: Engineering

Find the open pull request related to the
authentication timeout issue.

Expected tools:

search_pull_requests
get_pull_request

Task 3: Operations

Identify the active checkout latency incident
and retrieve its current service health.

Expected tools:

search_incidents
get_service_health

Task 4: Cross-System Workflow

Find the customer's latest failed order,
check whether an incident exists, and create
a support ticket if necessary.

Expected tools:

search_customer
search_orders
search_incidents
create_ticket

These tasks test both discovery and execution.

Measuring Tool Selection Accuracy

The first important metric is whether the agent selects the correct tool.

Define:

Tool Selection Accuracy =
Correct Tool Selections / Total Tool Selections

For example:

100 tool selections
92 correct
8 incorrect

Accuracy = 92%

But accuracy alone is not enough.

An agent may eventually find the correct tool after several incorrect searches.

Therefore, also measure:

First-Choice Accuracy

and:

Recovery Accuracy

These provide more detail.

First-Choice Accuracy

Suppose the correct tool is:

search_pull_requests

The agent chooses:

search_issues

first and then discovers the correct tool.

The final task may succeed, but the first-choice selection was incorrect.

This distinction matters in large tool catalogs.

First choice
     |
     X Wrong
     |
     v
Second search
     |
     v
Correct tool

Dynamic discovery should ideally reduce these errors by narrowing the candidate set.

Measuring Token Consumption

Tool definitions consume input context.

A static configuration might repeatedly send:

Tool A schema
Tool B schema
Tool C schema
...
Tool N schema

A dynamic configuration can instead keep only the relevant working set.

Microsoft states that Tool Search can keep the active tool context focused and reports that its Claude implementation can reduce tool-definition tokens substantially in large catalogs. These published figures are platform claims rather than an independent benchmark, so they should be validated against your own workload.

A benchmark should measure actual input tokens.

Static:
Input Tokens = Prompt + All Tool Definitions

Tool Search:
Input Tokens = Prompt + Search Interface + Retrieved Tools

Then calculate:

Token Reduction % =
(Static Tokens - Dynamic Tokens)
/
Static Tokens × 100

Measuring Latency

Tool Search introduces an additional discovery operation.

That creates a trade-off.

Static:

Request
  |
  v
Model
  |
  v
Tool

versus:

Tool Search:

Request
  |
  v
Model
  |
  v
Search
  |
  v
Tool

Dynamic discovery can reduce context size while adding search overhead.

Therefore, measure:

Initial model latency
Tool-search latency
Tool execution latency
Total task latency

A useful result table is:

Metric	Static	Tool Search
Median latency	Measure	Measure
p95 latency	Measure	Measure
Tool-search overhead	N/A	Measure
Input tokens	Measure	Measure
Failed selections	Measure	Measure

Do not assume Tool Search will always be faster.

The benchmark needs to determine the trade-off.

Measuring Tool-Selection Errors

A large tool catalog may contain tools with similar names.

For example:

search_issue
search_issues
search_pull_request
search_pull_requests
search_incident
search_incidents

The descriptions become extremely important.

Test cases should intentionally include similar tool names.

This helps determine whether the search mechanism can distinguish them reliably.

Tool Description Quality

Tool Search depends heavily on metadata quality.

Microsoft's documentation notes that tool names and descriptions influence discovery and recommends giving every tool a clear description. Vague or missing descriptions can produce poor search results.

Compare:

Name:
search_data

Description:
Search data.

with:

Name:
search_customer_orders

Description:
Find customer orders by customer ID,
order ID, status, or date range.
Use this when investigating customer
purchase history.

The second description provides substantially more semantic information.

Benchmarking Description Quality

Create two versions of the same toolbox.

Version A

Poor descriptions:

get_data
search_data
update_data

Version B

Intent-rich descriptions:

search_customer_orders
get_customer_order
update_order_status

Then compare Tool Search accuracy.

This separates two questions:

Does Tool Search work?

from:

Does Tool Search work with our tool metadata?

The second question is often more useful in production.

Pinning Critical Tools

Tool Search does not mean every tool must always be discovered dynamically.

Microsoft Foundry supports pinning important tools so they remain directly visible without a search round-trip.

Conceptually:

Always Available
 ├── get_current_user
 └── submit_response

Discovered Dynamically
 ├── GitHub tools
 ├── Monitoring tools
 ├── Database tools
 └── Customer tools

This hybrid strategy can be valuable when a small set of tools is used on almost every request.

Benchmarking Pinned vs Unpinned Tools

Add another experiment:

A:
Everything static

B:
Everything searchable

C:
Critical tools pinned
Everything else searchable

Then compare:

Configuration	Token Cost	Latency	Selection Accuracy
Static	Measure	Measure	Measure
Search	Measure	Measure	Measure
Hybrid	Measure	Measure	Measure

This is often more representative of a production architecture than a simple static-vs-search comparison.

Scaling the Tool Catalog

The most important experiment is to increase tool count.

For example:

At each level, run the same tasks.

Then plot:

Tool Count
     X
     |
     v
Token Usage
Selection Accuracy
Latency

The objective is to identify the point at which static loading becomes operationally expensive or unreliable for your workload.

Example Benchmark Dataset

A benchmark record could look like:

{
  "configuration": "tool-search",
  "toolCount": 250,
  "task": "support-incident",
  "firstChoiceCorrect": true,
  "finalToolCorrect": true,
  "toolSearches": 1,
  "toolCalls": 3,
  "inputTokens": 8200,
  "durationMs": 1840,
  "success": true
}

For static tools:

{
  "configuration": "static",
  "toolCount": 250,
  "task": "support-incident",
  "firstChoiceCorrect": false,
  "finalToolCorrect": true,
  "toolSearches": 0,
  "toolCalls": 4,
  "inputTokens": 38200,
  "durationMs": 2190,
  "success": true
}

These values are illustrative, not measured results.

The benchmark should populate the fields using actual traces.

Testing Large Enterprise Catalogs

Enterprise tool catalogs can be particularly challenging because different systems often use different naming conventions.

For example:

GitHub:
search_pull_requests

Monitoring:
query_incident_records

CRM:
find_customer_cases

Internal:
lookup_support_requests

A user may ask:

Find the customer's open support issue.

The tool names may not contain the exact phrase "support issue."

This is where additional_search_text can become useful.

Microsoft Foundry allows additional search text to be added to a tool's search index without changing the tool schema shown to the model.

For example:

Tool:
lookup_support_requests

Additional search terms:
customer issue, support ticket,
case, incident, complaint

This can help bridge differences between organizational vocabulary and tool names.

Failure Testing

The benchmark should also test when Tool Search returns the wrong tool.

For example:

User:
Find the open pull request for issue 4821.

If the agent receives:

search_issues

instead of:

search_pull_requests

measure what happens next.

A resilient agent should recognize the mismatch rather than repeatedly calling the wrong tool.

Test:

Correct first search
Wrong first search
No result
Multiple similar results
Ambiguous request

Benchmarking Ambiguous Requests

Real users rarely formulate perfect tool-search queries.

For example:

"Check the checkout problem."

could mean:

Search incidents
Search alerts
Check service health
Search customer tickets

A good benchmark should include ambiguous requests.

This tests whether the agent can use the discovered tool information and conversation context appropriately.

Common Mistakes

Assuming Fewer Tools Automatically Means Better Performance

Tool Search can reduce context, but discovery itself introduces work.

Measure the complete workflow.

Ignoring Tool Descriptions

Poor metadata can make a sophisticated search mechanism look ineffective.

Using Only Synthetic Tool Names

Real enterprise catalogs contain inconsistent naming conventions.

Use realistic metadata.

Measuring Only Token Savings

Lower token usage does not automatically mean higher task accuracy.

Ignoring Search Latency

A search round-trip has a cost.

Measure it.

Making Every Tool Dynamic

Some tools may be needed on nearly every request.

Pinning can avoid unnecessary discovery.

Troubleshooting Poor Tool Discovery

If Tool Search frequently returns the wrong tools:

Improve tool names.
Add precise descriptions.
Describe the intended use case.
Include important parameter terminology.
Add organization-specific search terms.
Pin frequently used critical tools.
Test searches against similar tool names.
Inspect the toolbox version being executed.

Microsoft recommends versioning toolboxes and testing a version-specific endpoint before promoting it as the default.

This is especially useful when changing tool descriptions or discovery configuration.

Best Practices

Keep Tool Descriptions Intent-Oriented

Describe both what the tool does and when it should be used.

Use Tool Search for Large Catalogs

The value increases as the tool catalog grows and different tasks use different subsets.

Pin High-Frequency Tools

Avoid a discovery round-trip for capabilities required on nearly every request.

Measure p50 and p95

Average latency can hide occasional expensive discovery or execution paths.

Track Tool-Selection Accuracy

Do not evaluate the feature only through token counts.

Version Your Toolbox

Treat tool definitions and discovery configuration as deployable artifacts.

Build Evaluation Tasks Around Real User Requests

Production queries are often ambiguous, incomplete, or expressed using internal terminology.

Advantages and Disadvantages

Advantages

Reduces the amount of tool metadata loaded into the active context.
Can improve tool selection in large catalogs.
Helps control input-token usage.
Allows tools to be discovered based on intent.
Supports pinning for frequently used capabilities.
Fits naturally with large enterprise toolboxes.

Disadvantages

Adds a discovery step.
Search quality depends heavily on tool metadata.
Ambiguous requests can still produce poor matches.
Requires additional evaluation and observability.
Tool availability and behavior may vary by Foundry feature version and deployment configuration.

Final Thoughts

The most interesting part of Tool Search is not simply that it can reduce the number of tool definitions sent to a model. It changes the architecture of tool-enabled agents from load everything first to discover capabilities when needed.

Microsoft Foundry's implementation uses tool_search and call_tool to keep the active tool set focused, while allowing the complete toolbox to remain available for discovery.

For Claude-based agents, this creates a measurable engineering trade-off. Static tools can be extremely effective when the catalog is small and predictable. Tool Search becomes more attractive as the catalog grows, particularly when different requests require different subsets of capabilities.

The right decision should therefore come from benchmarking your own workload.

Measure tool-selection accuracy, first-choice accuracy, input tokens, search latency, total latency, failed tool calls, and task success as the catalog scales from a handful of tools to hundreds or thousands.

That benchmark will tell you much more than a simple claim that dynamic tool discovery is "better." It will show exactly when it becomes better, why it becomes better, and where static tools may still be the more efficient choice.