AI applications are moving beyond simple request-and-response workflows.
A typical application may now have an agent that plans a task, calls tools, retrieves information, interacts with other agents, and continues working until the task is complete. Once these workloads run in production, the application needs more than an AI model endpoint.
It needs infrastructure for agents, tools, communication, security, state, observability, and workload scheduling.
Kubernetes is increasingly being used as the foundation for this type of architecture. Instead of building every piece of agent infrastructure independently, developers can place the agent runtime, supporting services, and workload controls on Kubernetes.
This article looks at how to design an AI application around an agent substrate on Kubernetes and what should be considered before moving the architecture into production.
What Is an Agent Substrate?
An agent substrate is the infrastructure layer that supports AI agents.
The agent itself is responsible for deciding what to do. The substrate provides the environment in which that agent can safely execute those actions.
A simplified architecture looks like this:
User
|
v
Application
|
v
Agent Runtime
|
+---- Model
|
+---- Tools
|
+---- Memory / State
|
+---- Other Agents
|
+---- External Services
|
v
Kubernetes
The important distinction is that Kubernetes is not the agent.
Kubernetes provides the platform for running and managing the services that make the agent application possible.
Why Kubernetes Fits Agent Workloads
Traditional web applications often have predictable request patterns.
An HTTP request arrives, the application processes it, and a response is returned.
Agent workloads can behave differently.
An agent may:
Receive a user request.
Create a plan.
Call a search tool.
Query a database.
Invoke another service.
Ask another agent to perform a task.
Wait for an asynchronous operation.
Retry a failed operation.
Continue processing.
Return the final result.
That creates a workload with more moving parts than a conventional API.
Kubernetes provides useful primitives for managing these components:
Containers
Services
Deployments
Jobs
ConfigMaps
Secrets
Horizontal scaling
Health checks
Network policies
Resource limits
Workload isolation
These capabilities make Kubernetes a practical substrate for complex AI systems.
Start With the Agent Boundary
Before creating Kubernetes resources, define what the agent actually owns.
For example, an application might contain:
Customer Support Application
|
v
Support Agent
|
+-- Search Knowledge Base
+-- Check Order
+-- Create Support Ticket
+-- Send Notification
The agent should not automatically receive unrestricted access to every backend service.
Instead, each capability should have a clear interface.
For example:
Agent
|
+-- searchKnowledgeBase()
+-- getOrderStatus()
+-- createSupportTicket()
This makes the system easier to secure and test.
It also prevents the agent from becoming a large application layer with direct access to internal infrastructure.
A Practical Kubernetes Architecture
A production-oriented architecture can separate the application into several workloads.
Users
|
v
API Gateway
|
v
Agent Orchestrator
|
+-----------+-----------+
| | |
v v v
Tool API Agent API Memory API
| | |
+-----------+-----------+
|
v
External Services
Kubernetes Cluster
The cluster may contain:
The exact number of services depends on the application.
Do not split every function into a separate microservice simply because Kubernetes makes it possible.
Separate Agent Logic From Tool Execution
One of the most important design decisions is keeping agent reasoning separate from tool execution.
For example:
Agent
|
| requests:
| "Get customer order status"
v
Tool Service
|
v
Order Database
The agent should receive a controlled result rather than direct database credentials.
A tool service might expose:
POST /tools/order-status
with a request such as:
{
"orderId": "ORD-10045"
}
The service validates the request and retrieves the information.
This creates a security boundary between the agent and the underlying infrastructure.
Use Kubernetes Services for Internal Communication
Agent workloads frequently need to communicate with other services.
Kubernetes Services provide stable internal endpoints even when pods are replaced.
For example:
agent-service
tool-service
memory-service
model-gateway
The agent application can communicate with:
http://tool-service
instead of depending on a specific pod IP address.
This is important because pods are temporary.
A pod can be restarted, rescheduled, or replaced without changing the application's service-level architecture.
Handle Long-Running Agent Tasks
Not every agent operation should remain inside a single HTTP request.
Consider an agent that performs a long-running task:
User Request
|
v
Create Agent Job
|
v
Queue
|
v
Agent Worker
|
+-- Tool Call
+-- Data Processing
+-- Model Call
+-- Validation
|
v
Result
For these workloads, asynchronous processing is often more appropriate than keeping an HTTP connection open for several minutes.
Kubernetes Jobs or long-running worker Deployments can execute the workload depending on its lifecycle.
The application can return a job identifier:
{
"taskId": "task-78431",
"status": "running"
}
The client can then retrieve the status later.
This also makes retries and recovery easier.
Manage Agent State Carefully
Agents often need state.
Examples include:
Do not assume Kubernetes pod memory is a reliable place for persistent agent state.
A pod can disappear at any time.
Instead, separate temporary execution state from durable application state.
For example:
Agent Pod
|
+-- Temporary execution context
|
v
State Store
|
+-- Conversation state
+-- Workflow state
+-- Task metadata
This allows another agent worker to continue the task if the original pod is replaced.
Scaling Agent Workloads
Traditional CPU-based autoscaling is not always enough for AI workloads.
An agent service may be lightly loaded while its model requests are expensive or slow.
You should monitor signals such as:
Request rate
Queue depth
Request latency
Active agent sessions
Model request latency
CPU utilization
Memory utilization
GPU utilization
Tool execution latency
For asynchronous workers, queue depth can be particularly useful.
For example:
Queue depth = 10
|
v
2 workers
Queue depth = 500
|
v
20 workers
The exact scaling policy depends on workload behavior.
The important point is to scale based on the bottleneck rather than relying only on CPU utilization.
Resource Requests and Limits
Agent services should define Kubernetes resource requirements.
For example:
resources:
requests:
cpu: "500m"
memory: "512Mi"
limits:
cpu: "2"
memory: "2Gi"
This prevents a poorly behaved agent workload from consuming the entire node.
For model-serving workloads, GPU resources may also need to be declared.
Resource planning becomes especially important when several agent workloads share the same cluster.
Treat Tool Access as a Security Boundary
An AI agent should not automatically have access to every internal service.
Instead, use an allow-list approach.
For example:
Support Agent
|
+-- Knowledge Search Allowed
+-- Order Service Allowed
+-- Ticket Service Allowed
+-- Payroll Database Denied
+-- Production Shell Denied
Kubernetes NetworkPolicies can help restrict network communication between workloads.
Application-level authorization should still be implemented.
Network isolation alone does not provide complete security.
Protect Secrets
Agent workloads may need credentials for:
Model providers
Databases
APIs
Cloud services
SaaS platforms
Do not place these credentials directly in container images or source code.
Kubernetes Secrets can be used as part of the solution, although production environments may also integrate with an external secrets-management system.
A deployment might reference a secret:
env:
- name: API_KEY
valueFrom:
secretKeyRef:
name: model-api-secret
key: api-key
The application receives the credential without storing it in source control.
Add Guardrails Before Tool Execution
Agent autonomy introduces a different security problem.
Traditional applications control the exact sequence of operations.
An agent can choose the next tool based on its reasoning.
That means tool execution should have explicit controls.
For example:
Agent Decision
|
v
Policy Check
|
+---- Denied ----> Stop / Request Approval
|
v
Tool Execution
|
v
Result Validation
The policy layer can check:
User authorization
Tool permissions
Input validation
Data sensitivity
Request limits
Approval requirements
Allowed destinations
This becomes especially important for tools that modify production data.
Observability for Agent Applications
Traditional logs are not enough for complex agent systems.
A production troubleshooting workflow may need to answer:
Which agent started the task?
Which model was used?
Which tools were called?
How long did each call take?
Which call failed?
How many retries occurred?
What was the final result?
Use a correlation ID across the complete execution.
For example:
Request ID: req-9012
Agent
|
+-- Model Call 420 ms
|
+-- Search Tool 180 ms
|
+-- Order Tool 240 ms
|
+-- Model Call 610 ms
|
+-- Final Response
Distributed tracing is particularly useful when agent execution crosses multiple Kubernetes services.
Handle Model Failures Separately From Application Failures
A model request can fail for reasons that have nothing to do with Kubernetes.
Examples include:
The application should distinguish these failures from infrastructure failures.
For example:
Model Timeout
|
+-- Retry if safe
+-- Use fallback model
+-- Queue task
+-- Return controlled error
Do not blindly retry every model request.
If an agent already performed a write operation, retrying the entire workflow may create duplicate actions.
Idempotency Matters
Consider an agent that creates a support ticket.
The workflow is:
Agent
|
v
Create Ticket
|
v
Timeout
The agent does not know whether the ticket was actually created.
If it simply retries:
Create Ticket
Create Ticket
the customer may receive duplicate tickets.
Use idempotency keys where supported.
For example:
Idempotency-Key:
task-78431-create-ticket
The downstream service can recognize repeated requests.
This is an important production requirement for autonomous workflows.
Version Agent Workflows
Agent behavior changes as prompts, models, tools, and policies change.
Do not treat the agent as an unversioned application component.
Version at least:
Agent workflow
Tool contracts
Prompt configuration
Model selection
Security policies
Evaluation datasets
For example:
Agent v1
Tool schema v1
Policy v1
Model A
Agent v2
Tool schema v2
Policy v2
Model B
This makes it possible to roll back a change when an updated agent starts producing unexpected results.
Common Design Mistakes
Giving Agents Direct Database Access
Agents should normally interact through controlled tools and services.
Keeping State Only Inside Pods
Pods are replaceable. Durable state belongs in an appropriate persistent system.
Scaling Only on CPU
AI workloads can be constrained by model latency, GPU capacity, or queue depth rather than CPU.
Allowing Every Agent to Use Every Tool
Use explicit tool permissions.
Logging Only the Final Answer
The final answer does not explain how an agent reached the result.
Capture the execution path and important tool interactions.
Retrying Complete Workflows Blindly
Retries can duplicate side effects.
Use idempotency and retry only operations that are safe to repeat.
Advantages of an Agent Substrate on Kubernetes
This architecture can provide:
Standardized workload management
Container-based isolation
Flexible scaling
Service discovery
Network controls
Centralized observability
Integration with existing cloud infrastructure
Support for asynchronous workloads
Clear separation between agent logic and supporting services
It also allows platform teams to create common infrastructure that multiple AI applications can share.
Limitations and Trade-Offs
Kubernetes also adds operational complexity.
Teams may need to manage:
Cluster upgrades
Networking
Security policies
Resource allocation
Observability
Autoscaling
GPU scheduling
Secrets
Cost management
A small application may not need this level of infrastructure.
The architecture becomes more useful when the organization has multiple AI workloads, complex agent workflows, strict security requirements, or significant operational needs.
A Practical Production Checklist
Before deploying an agent-based application to Kubernetes, verify:
[ ] Agent responsibilities are clearly defined
[ ] Tools have explicit interfaces
[ ] Tool permissions are restricted
[ ] Agent state is stored appropriately
[ ] Long-running tasks use asynchronous processing where needed
[ ] Kubernetes resource requests and limits are configured
[ ] Autoscaling signals are defined
[ ] Network policies are configured
[ ] Secrets are protected
[ ] Agent actions have policy checks
[ ] Correlation IDs are implemented
[ ] Distributed tracing is available
[ ] Model failures are handled separately
[ ] Retry behavior is safe
[ ] Idempotency is implemented for side effects
[ ] Agent workflows are versioned
[ ] Model and tool changes can be rolled back
[ ] Monitoring and alerts are configured
[ ] Load testing is complete
[ ] Disaster recovery has been tested
Final Thoughts
Building an AI application on Kubernetes is not simply a matter of putting an agent inside a container.
The more useful approach is to treat Kubernetes as the substrate for a collection of controlled capabilities: agent execution, tools, state, model access, asynchronous workers, security, and observability.
The agent should decide what needs to happen, while the surrounding infrastructure controls what the agent is allowed to do and provides the reliability needed to execute those actions.
This separation becomes increasingly important as AI applications move from experimental chat interfaces to production systems that can perform real work.
A well-designed Kubernetes-based agent architecture should therefore focus on clear tool boundaries, durable state, controlled permissions, observable execution, safe retries, and versioned workflows.
Those foundations make it easier to scale an AI application without turning the agent itself into an uncontrolled access point to the rest of the system.
Join the conversation! Your thoughts help the community grow.