Microsoft Foundry Hosted Agents: Benchmarking Sandbox Startup and Isolation Overhead

Introduction

AI agents are increasingly expected to do more than generate text. They may inspect files, execute code, run tests, call utilities, transform data, and interact with application environments.

That creates an important infrastructure question:

Where should an agent execute its work?

Running agent workloads directly inside the application process is simple, but it creates security and isolation concerns. Running every task inside a dedicated sandbox provides stronger boundaries, but introduces startup and resource overhead.

Hosted agent environments are designed to address this problem by providing an execution environment where agent workloads can run with controlled isolation.

For engineering teams, however, security is only one part of the decision. A production system also needs to understand the performance characteristics of that isolation.

The most important questions are:

How long does a sandbox take to become ready?
How much latency does startup add?
Does a warm environment behave differently from a cold environment?
How much memory and CPU does isolation consume?
How does startup overhead affect short-lived tasks?
Does stronger isolation reduce throughput?
How does concurrency change the results?

This article explains how to benchmark sandbox startup and isolation overhead for hosted AI agents and how to interpret the results without confusing infrastructure overhead with model performance.

What Is a Hosted Agent Environment?

A hosted agent environment can be thought of as a controlled execution boundary around an agent workload.

Conceptually:

User Request
     |
     v
Agent
     |
     v
Hosted Execution Environment
     |
     +--> Files
     +--> Tools
     +--> Commands
     +--> Runtime
     +--> Network Policy
     |
     v
Task Result

Instead of executing potentially risky operations directly inside the primary application process, the workload runs inside an isolated environment.

The exact implementation can vary, but the architectural goal is generally the same:

Agent Workload
      |
      v
Isolation Boundary
      |
      v
Controlled Execution

This separation is particularly useful when an agent needs to execute code or interact with resources that should not have unrestricted access to the host application.

Why Startup Overhead Matters

Sandboxing introduces infrastructure work before the actual task begins.

A simplified lifecycle looks like this:

Request
   |
   v
Allocate Environment
   |
   v
Initialize Runtime
   |
   v
Prepare Workspace
   |
   v
Load Required Resources
   |
   v
Environment Ready
   |
   v
Agent Task

For a long-running task, an additional 1–2 seconds of startup may be insignificant.

For a short task, it can dominate the total response time.

Consider an illustrative example:

Task Execution: 2 seconds
Sandbox Startup: 1 second

Total: 3 seconds

Startup represents one-third of the total elapsed time.

For another workload:

Task Execution: 60 seconds
Sandbox Startup: 1 second

Total: 61 seconds

The same startup overhead is now much less significant.

Therefore, sandbox overhead should always be evaluated relative to task duration.

Cold Start vs Warm Start

The first benchmark distinction should be between cold and warm execution.

Cold Start

A cold start means the execution environment is not already available.

Request
  |
  v
Create Environment
  |
  v
Initialize
  |
  v
Execute Task

This represents the highest startup overhead.

Warm Start

A warm environment can reduce initialization work.

Existing Environment
        |
        v
Execute Task

A benchmark that mixes cold and warm executions without recording the state can produce misleading averages.

Always classify each execution.

Benchmark Lifecycle Phases

Instead of measuring only total latency, break the execution into phases.

T0
 |
 +--> Environment Allocation
 |
 +--> Runtime Initialization
 |
 +--> Workspace Preparation
 |
 +--> Tool Initialization
 |
 +--> Agent Execution
 |
 +--> Cleanup
 |
T1

A useful metric model is:

Total Time =
Startup Time
+ Initialization Time
+ Agent Execution Time
+ Cleanup Time

This decomposition helps identify the actual source of latency.

Core Benchmark Metrics

At minimum, record:

Metric	Description
Cold startup	Time to create a fresh environment
Warm startup	Time to reuse an available environment
Ready latency	Time until execution can begin
Task latency	Time spent executing the task
Total latency	End-to-end duration
CPU usage	CPU consumed during execution
Memory usage	Memory footprint
Failure rate	Failed environment/task starts
Cleanup time	Time required to release resources

For production analysis, also capture:

P50
P95
P99
Maximum observed latency

Average startup time alone is not sufficient.

Why Percentiles Matter

Suppose 100 sandbox starts produce:

Average: 900 ms
P50:     700 ms
P95:     1.8 s
P99:     4.5 s

The average looks reasonable.

However, 1% of requests taking several seconds may be significant for an interactive agent.

This is why infrastructure benchmarks should focus heavily on tail latency.

Designing a Controlled Benchmark

The benchmark should use repeatable tasks.

For example:

Task A:
Read a small file and return its contents.

Task B:
Search several files and summarize matching code.

Task C:
Modify a file and run a small test.

Task D:
Build a medium-sized .NET project.

Task E:
Run a complete test suite.

These tasks represent different execution durations.

The benchmark should execute each task under controlled conditions.

Establish a Baseline

Before measuring hosted sandbox performance, measure the same workload without the sandbox boundary where it is safe and technically appropriate.

For example:

Baseline
Application
   |
   v
Task

Hosted
Application
   |
   v
Sandbox
   |
   v
Task

The baseline gives you an estimate of the infrastructure overhead.

The comparison should not be interpreted as a recommendation to remove isolation. The purpose is to quantify what the isolation boundary costs.

Calculate Isolation Overhead

A simple measurement is:

Isolation Overhead =
Hosted Execution Time
-
Baseline Execution Time

You can also calculate relative overhead:

Relative Overhead =
(Hosted - Baseline)
-------------------
Baseline
× 100

For example, using illustrative values:

Baseline:  2.0 seconds
Hosted:    2.5 seconds

Overhead:
0.5 seconds

Relative:
25%

The absolute and relative measurements tell different stories.

Benchmark Harness in .NET

A simple C# model can capture the important measurements.

public sealed record SandboxBenchmarkResult(
    string TaskId,
    string ExecutionMode,
    bool ColdStart,
    long StartupMs,
    long ExecutionMs,
    long CleanupMs,
    long TotalMs,
    long PeakMemoryBytes,
    bool Succeeded);

The benchmark runner can then execute multiple samples and aggregate them.

var grouped = results
    .GroupBy(x => new
    {
        x.TaskId,
        x.ExecutionMode,
        x.ColdStart
    })
    .Select(group => new
    {
        group.Key.TaskId,
        group.Key.ExecutionMode,
        group.Key.ColdStart,
        Samples = group.Count(),
        AverageStartupMs = group.Average(x => x.StartupMs),
        AverageExecutionMs = group.Average(x => x.ExecutionMs),
        AverageTotalMs = group.Average(x => x.TotalMs),
        SuccessRate = group.Average(x => x.Succeeded ? 1.0 : 0.0)
    });

This provides a useful starting point for comparing execution modes.

Measuring Startup Precisely

Do not measure startup from the wrong point.

For example:

Wrong:
Request received -> final response

Better:
Environment creation -> environment ready

The benchmark should define clear timestamps:

var allocationStart = Stopwatch.GetTimestamp();

await AllocateEnvironmentAsync();

var environmentReady = Stopwatch.GetTimestamp();

await ExecuteTaskAsync();

var taskCompleted = Stopwatch.GetTimestamp();

Then calculate separate durations.

This prevents model execution time from being accidentally included in the startup metric.

Task Duration Matters

Consider two tasks:

Task A
Startup:   1.0s
Execution: 0.5s
Total:     1.5s

and:

Task B
Startup:   1.0s
Execution: 30s
Total:     31s

The same infrastructure overhead has dramatically different significance.

A good benchmark should therefore report:

Startup / Total Task Time

as well as absolute startup latency.

Measuring Memory Overhead

Isolation can introduce additional memory consumption.

Measure memory at multiple points:

Before Environment
       |
       v
After Environment Ready
       |
       v
During Task
       |
       v
After Cleanup

Conceptually:

Isolation Memory Overhead =
Peak Hosted Memory
-
Peak Baseline Memory

Avoid treating process working-set measurements as exact application memory consumption. Runtime behavior, filesystem caching, shared resources, and garbage collection can affect the result.

Measuring CPU Overhead

CPU should also be measured separately from task CPU consumption.

For example:

Environment Initialization
        |
        v
CPU Usage
        |
        v
Task Execution
        |
        v
Cleanup

If sandbox initialization consumes significant CPU, high-concurrency workloads may encounter resource contention before the actual agent work begins.

Concurrency Changes the Result

A single sandbox benchmark does not represent production behavior.

Consider:

1 request
   |
   v
1 sandbox

versus:

100 requests
   |
   +--> Sandbox
   +--> Sandbox
   +--> Sandbox
   ...

At higher concurrency, you may encounter:

CPU contention
Memory pressure
Environment allocation delays
Queueing
Startup latency increases
Resource exhaustion
Higher failure rates

Therefore, benchmark concurrency explicitly.

A useful test matrix is:

Concurrency	Cold Starts	Warm Starts
1	Test	Test
5	Test	Test
10	Test	Test
25	Test	Test
50	Test	Test
100	Test	Test

The appropriate range depends on the expected production workload.

Warm Pool Analysis

If environments can be reused, a warm pool can potentially reduce startup latency.

Conceptually:

             +--> Agent A
Pool ------> +--> Agent B
             +--> Agent C

Instead of creating an environment for every request:

Request
  |
  v
Create
  |
  v
Execute
  |
  v
Destroy

the system can potentially reuse prepared capacity:

Prepared Environment
        |
        v
      Task
        |
        v
     Reuse

However, reuse introduces its own concerns.

The benchmark should evaluate:

Pool size
Idle timeout
Reuse rate
Warm hit rate
Memory consumption
Cleanup behavior
Isolation between sequential tasks

Isolation Must Survive Reuse

A warm environment should never imply that state from one task becomes available to another task unintentionally.

Consider:

Task A
 |
 +--> Creates file
 +--> Sets environment variable
 +--> Writes temporary data
 |
 v
Environment Reused
 |
 v
Task B

Task B should not unexpectedly inherit Task A's state.

Therefore, isolation testing should include cross-task contamination tests.

For example:

Task A:
Create /workspace/secret.txt

Task B:
Attempt to read /workspace/secret.txt

The expected result should be explicitly defined by the environment's isolation model.

Network Isolation

Network access is another major benchmark dimension.

Test:

No Network
Limited Network
Required Internal Services
External Services

Measure both behavior and latency.

A network-restricted environment might fail quickly, while an environment attempting an unavailable connection may experience connection timeouts.

Those differences can materially affect agent task latency.

Failure Scenarios

A good benchmark should deliberately test failures.

Examples include:

Environment allocation failure
Runtime initialization failure
Tool startup failure
Network timeout
Memory exhaustion
CPU exhaustion
Task timeout
Cleanup failure

Record:

Failure Rate
Recovery Time
Retry Count
Final Task Success

A fast successful environment is useful, but an environment that fails unpredictably under load can become a production problem.

Benchmarking Short vs Long Tasks

Divide tasks into duration categories.

Short:
< 5 seconds

Medium:
5–30 seconds

Long:
> 30 seconds

The exact boundaries should be adjusted to the workload.

Then compare relative sandbox overhead.

This helps answer an important architecture question:

Is the isolation overhead acceptable for the workload we actually have?

A platform optimized for long-running agent tasks may have very different characteristics from one optimized for sub-second interactions.

Example Benchmark Results

The following values are illustrative rather than production measurements.

Task	Baseline	Cold Hosted	Warm Hosted
Read file	0.4s	1.5s	0.6s
Search repository	1.8s	2.9s	2.0s
Edit + test	4.5s	5.8s	4.9s
Build project	15s	16.8s	15.9s
Full test suite	42s	44s	43s

The important observation is not the specific numbers.

The benchmark demonstrates how the same startup cost can have very different relative impact depending on the task.

Common Benchmarking Mistakes

Measuring Only Average Startup

Tail latency can be much more important than the mean.

Mixing Cold and Warm Runs

Always label environment state.

Using Only One Task

A single task cannot characterize sandbox behavior.

Ignoring Concurrency

A system that performs well at concurrency one may behave differently under production load.

Including Model Latency in Startup Metrics

Separate infrastructure startup from agent execution.

Ignoring Cleanup

Cleanup can consume resources and affect throughput.

Reusing Environments Without Testing State Isolation

Warm execution must not compromise isolation.

Measuring Only CPU

Memory, network, storage, and allocation delays can also affect performance.

Best Practices

Separate cold-start and warm-start benchmarks.
Measure startup, execution, and cleanup independently.
Report P50, P95, and P99 latency.
Use multiple representative task types.
Benchmark different task durations.
Test realistic concurrency levels.
Measure CPU and memory overhead.
Test network restrictions explicitly.
Validate isolation between sequential tasks.
Include failure and timeout scenarios.
Measure warm-pool effectiveness when applicable.
Record environment configuration with every benchmark result.
Compare against a clearly defined baseline.
Repeat tests enough times to account for infrastructure variability.
Optimize only after identifying the actual bottleneck.

Frequently Asked Questions

Is sandbox startup overhead always a problem?

No. For long-running or security-sensitive workloads, a modest startup cost may be an acceptable tradeoff for stronger isolation.

Should every agent task use a fresh environment?

Not necessarily. The right strategy depends on the required isolation guarantees, workload duration, startup characteristics, and whether safe reuse is supported.

Why should cold and warm execution be benchmarked separately?

They represent fundamentally different infrastructure paths. Mixing them can hide the actual startup characteristics.

What is more important: average latency or P95?

For production interactive systems, P95 and P99 often provide more useful information because they expose tail behavior.

Does a warm environment eliminate isolation overhead?

No. It may reduce initialization latency, but resource consumption and isolation mechanisms still exist.

Should model latency be included in the benchmark?

Yes for end-to-end task latency, but it should be reported separately from environment startup so the two effects can be analyzed independently.

Conclusion

Hosted execution environments provide an important isolation boundary for AI agents that need to perform potentially sensitive or untrusted operations. But isolation has an infrastructure cost, and that cost should be measured rather than assumed.

A useful benchmark separates cold startup, warm startup, task execution, cleanup, resource consumption, concurrency, and failure behavior. It also compares short and long-running tasks because startup overhead has a very different impact on each.

The goal is not to eliminate sandbox overhead at any cost. The goal is to understand its contribution to the overall agent workload and determine whether the security and isolation benefits justify that cost for the target application.

For production AI systems, the right question is therefore not simply "How fast can the agent run?" It is "How much infrastructure overhead does secure, isolated execution add to a successfully completed agent task under realistic load?"