Running sensitive or untrusted workloads in an isolated environment is one thing. Proving that the isolation works as expected before production is another.
A sandbox can look secure from an architecture diagram and still have problems in areas such as networking, resource limits, startup behavior, workload cleanup, logging, or application integration.
AWS Nitro-based isolation technologies are designed to provide strong workload isolation, but production testing should still be treated as a separate engineering exercise. The goal is not simply to verify that a sandbox starts. The goal is to understand how the workload behaves when it is given limited resources and when it encounters conditions that are common in production.
This article walks through a practical testing approach for Nitro-based sandbox workloads before using them for sensitive applications.
Start With a Threat Model
Before testing the sandbox itself, define what you are trying to protect.
For example, suppose an application allows users to execute code:
Client
|
v
API
|
v
Execution Service
|
v
Nitro-based Sandbox
|
v
Untrusted Workload
The workload may need access to input data, temporary storage, or a specific service.
It should not automatically have access to:
- Application secrets
- Production databases
- Other tenants' data
- Internal administration services
- Unrestricted network resources
- Host resources
- Credentials belonging to the main application
Write these requirements down before starting the test.
A useful test plan should answer questions such as:
Can the workload access anything outside its intended boundary?
What happens when the workload consumes too much memory?
What happens when it runs for too long?
What happens when the network is unavailable?
What happens when the sandbox is terminated?
Can data from one job appear in another job?
Can the application safely recover from a failed execution?
These questions are more useful than simply checking whether the sandbox launches successfully.
Separate Functional Testing From Security Testing
A common mistake is to treat a successful application test as proof of isolation.
It is not.
Functional testing asks:
Does the workload produce the expected result?
Security testing asks:
Can the workload do something it should not be able to do?
Both need to be tested independently.
For example, a normal test might execute:
print("Hello from sandbox")
A security-oriented test should attempt operations that are explicitly outside the workload's permissions.
The objective is not to create a production exploit. It is to verify that prohibited operations are rejected and that the application handles those failures correctly.
Build a Test Matrix
Instead of running a few manual tests, create a repeatable test matrix.
| Test area | What to verify |
|---|
| Startup | Sandbox starts within the expected time |
| Isolation | Workload cannot access prohibited resources |
| IAM | Only required permissions are available |
| Network | Only approved destinations are reachable |
| CPU | CPU limits are enforced |
| Memory | Memory limits are enforced |
| Storage | Temporary storage behaves as expected |
| Timeout | Long-running jobs are terminated |
| Cleanup | Temporary data disappears after execution |
| Logging | Execution details are captured safely |
| Failure | Application handles sandbox failures |
| Concurrency | Multiple jobs remain isolated |
| Recovery | Failed jobs can be retried safely |
This matrix can become part of the deployment process instead of remaining as a one-time security checklist.
Test the Sandbox With Minimal Permissions
Start with the smallest possible execution role.
For example, if a workload needs to read one S3 object, give it access only to that required resource.
A simplified policy could look like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:GetObject",
"Resource": "arn:aws:s3:::sandbox-inputs/jobs/*"
}
]
}
Then test both sides of the permission boundary.
The expected behavior is:
Allowed resource
|
+-- Read succeeds
Unexpected resource
|
+-- Access denied
This verifies that the workload is operating with the permissions you intended rather than the permissions that happen to be available in the environment.
Test Network Isolation
Network testing deserves special attention because an isolated workload can still become dangerous if it has broad network connectivity.
Start by defining exactly what the workload needs.
For example:
Sandbox
|
+-- Object storage Allowed
|
+-- Required API Allowed
|
+-- Production database Blocked
|
+-- Admin service Blocked
|
+-- Unnecessary Internet Blocked
The test should verify both allowed and denied paths.
Do not only test that the required endpoint works. Test that unnecessary endpoints are inaccessible.
This creates a much clearer picture of the actual network boundary.
Test Resource Exhaustion
A sandbox should be tested under resource pressure.
For example, create workloads that consume increasing amounts of memory:
128 MB
256 MB
512 MB
1 GB
Observe where the configured limit is enforced and how the execution service reports the failure.
Do the same for CPU-intensive workloads.
A simple CPU stress test might look like:
total = 0
while True:
total += 1
The purpose is not to crash infrastructure. The purpose is to verify that the sandbox's resource controls behave predictably.
The application should receive a controlled failure rather than becoming unstable because one job consumed resources indefinitely.
Test Execution Timeouts
Timeout behavior should also be tested explicitly.
Consider a job that never exits:
import time
while True:
time.sleep(1)
If the execution service has a 30-second limit, the job should be terminated according to the platform's configured behavior.
The application should then record something similar to:
Job ID: 8f31c2
Status: TimedOut
Duration: 30 seconds
Retryable: No
The exact status model depends on the application, but the important point is that timeout behavior should be deterministic.
Do not leave timeout handling until production.
Test Temporary Data Cleanup
Sensitive workloads often create temporary files.
For example:
/input/data.json
/output/result.json
/tmp/intermediate.bin
After the job completes, verify that temporary data is not accidentally reused by another execution.
A useful test sequence is:
Job A
|
+-- Write unique marker
|
+-- Complete
|
v
Sandbox cleaned
Job B
|
+-- Search for Job A marker
|
+-- Must not find it
This is particularly important in multi-tenant systems.
Each execution should be treated as an independent security boundary.
Test Concurrent Executions
A sandbox architecture may work correctly with one workload and behave differently under concurrency.
Run several independent jobs at the same time:
Job A ──> Sandbox A
Job B ──> Sandbox B
Job C ──> Sandbox C
Job D ──> Sandbox D
Then verify:
- CPU allocation
- Memory allocation
- Startup latency
- Network behavior
- Logging
- Temporary storage
- Job identification
- Cleanup
Also test workloads belonging to different users or tenants.
The objective is to ensure that concurrency does not accidentally introduce shared state.
Test Failure Handling
Production systems need to assume that sandbox execution will sometimes fail.
Test conditions such as:
- Sandbox startup failure
- Workload crash
- Timeout
- Permission denial
- Network failure
- Invalid input
- Resource limit exceeded
- Unexpected process termination
The application should distinguish between different failure types.
For example:
Validation failure
|
+-- Do not retry
Temporary infrastructure failure
|
+-- Retry if safe
Resource limit exceeded
|
+-- Return controlled failure
Application error
|
+-- Record diagnostic information
This prevents the application from blindly retrying workloads that are guaranteed to fail.
Test Retry and Idempotency
Retries can create another problem.
Suppose a sandbox job performs an operation and then the execution service reports a timeout.
The application may not know whether the workload completed before the timeout was recorded.
If the application retries the job, the operation could run twice.
For this reason, sensitive workloads should have a clear job identity.
For example:
Job ID: 7d4f21
Attempt: 1
Status: Timeout
A retry can then become:
Job ID: 7d4f21
Attempt: 2
The underlying operation should be designed to avoid unintended duplicate effects.
This matters especially when the sandbox is allowed to call external APIs or modify persistent data.
Test Observability
A production-ready sandbox should provide enough information to answer:
Who started the job?
When did it start?
Which sandbox handled it?
How long did it run?
How much CPU and memory did it consume?
Why did it stop?
What was the final status?
A useful execution record might contain:
{
"jobId": "7d4f21",
"status": "Completed",
"durationMs": 8420,
"exitCode": 0,
"attempt": 1
}
Avoid storing sensitive input or secrets in logs simply because the execution system makes logging convenient.
Observability should help operators diagnose problems without becoming another source of data exposure.
Automate the Security Tests
Manual testing is useful during initial development, but it is not enough for a production system.
The most important tests should eventually run automatically.
For example:
Pull Request
|
v
Build Sandbox Image
|
v
Run Isolation Tests
|
v
Run Permission Tests
|
v
Run Network Tests
|
v
Run Resource Tests
|
v
Publish Result
This gives the team an early warning when changes to the runtime, IAM policy, networking configuration, or execution service alter the security boundary.
Security tests should be treated as regression tests.
Test Changes to the Sandbox Environment
The sandbox itself is only one part of the system.
A production change may involve:
- Container image changes
- IAM policy changes
- Network rule changes
- Runtime updates
- Resource configuration
- Application code
- Logging configuration
Each change can affect the overall security model.
For this reason, maintain a baseline test suite.
For example:
Baseline
|
+-- Isolation
+-- IAM
+-- Network
+-- Resource limits
+-- Cleanup
+-- Failure handling
Run the same tests after significant infrastructure changes.
Common Testing Mistakes
Testing Only Successful Workloads
A sandbox that successfully runs normal code has not necessarily been security tested.
Include workloads that exceed limits, request unavailable permissions, and attempt prohibited network access.
Testing Only One Job
Isolation problems may appear only under concurrency.
Test multiple jobs and multiple tenants.
Ignoring Cleanup
A workload that completes successfully may still leave sensitive data behind.
Always test the lifecycle after execution.
Treating IAM as an Application Concern
The application and sandbox should have separate permissions.
Test the execution role independently.
Logging Everything
Verbose logs can accidentally expose secrets or user data.
Define what should and should not be logged before enabling production logging.
Production Readiness Checklist
Before allowing sensitive workloads to use the sandbox, verify:
| Area | Production check |
|---|
| Threat model | Security boundary is documented |
| IAM | Dedicated least-privilege role |
| Network | Only required destinations are reachable |
| Isolation | Unauthorized resource access is blocked |
| Resource limits | CPU and memory limits are enforced |
| Timeout | Long-running workloads terminate correctly |
| Cleanup | Temporary data does not survive unexpectedly |
| Concurrency | Multiple workloads remain isolated |
| Failure handling | Sandbox failures are handled safely |
| Retry | Jobs have clear identity and retry behavior |
| Logging | Sensitive information is excluded |
| Monitoring | Resource and execution behavior is observable |
| Regression tests | Security tests run after infrastructure changes |
Summary
Testing a Nitro-based sandbox should go beyond confirming that a workload can start and finish successfully.
The important question is whether the complete execution boundary behaves correctly when the workload is slow, resource-intensive, network-dependent, unauthorized, or simply fails.
Start with a clear threat model, test IAM and network boundaries, enforce resource and execution limits, verify cleanup, test concurrent workloads, and automate the security checks.
Once these tests become part of the deployment process, sandbox isolation becomes something the team continuously verifies rather than something it assumes is working.
Join the conversation! Your thoughts help the community grow.