Running an AI model locally is easy when the goal is only to test a model.
Running the same model as a shared service is a different problem.
Once multiple applications need inference at the same time, you have to think about model loading, GPU allocation, request routing, scaling, health checks, observability, and what happens when one inference worker becomes unavailable.
Kubernetes can provide the infrastructure layer for this, but simply putting a model server inside a Deployment does not automatically create a good AI inference platform.
AI inference has characteristics that normal web applications do not have. Requests can be long-running, GPU memory is limited, model weights can consume several gigabytes or more, and performance can depend heavily on which model replica receives a request.
Kubernetes is increasingly adding workload and networking capabilities that are useful for these scenarios. Gateway API's Inference Extension, for example, introduces inference-aware routing concepts such as InferencePool and InferenceModel instead of treating every AI request like ordinary HTTP traffic.
This article builds a small conceptual inference platform around Kubernetes and shows where the newer AI-specific pieces fit.
What an AI Inference Cluster Actually Does
An inference cluster receives a request from an application and sends it to a model server running on compute hardware such as a GPU node.
A simple architecture looks like this:
Client Application
|
v
API Gateway
|
v
Inference Router
|
+----------------+
| |
v v
Model Server A Model Server B
| |
v v
GPU A GPU B
The model servers may run the same model or different models.
For example:
Model A
Llama-based chat model
|
+-- GPU Node 1
+-- GPU Node 2
Model B
Embedding model
|
+-- GPU Node 3
The Kubernetes cluster manages the infrastructure around these model servers.
It can schedule Pods, assign resources, restart failed containers, expose Services, manage configuration, and coordinate scaling.
The model server itself remains responsible for loading the model and processing inference requests.
This separation is important.
Kubernetes is not the model runtime.
It is the platform that manages the workloads running the model runtime.
Why a Normal Kubernetes Service Is Not Always Enough
A normal Kubernetes Service distributes traffic across available Pods.
For a conventional HTTP application, this is often sufficient.
For example:
Request 1 ---> Pod A
Request 2 ---> Pod B
Request 3 ---> Pod C
AI inference is more complicated.
Imagine three model servers:
Pod A
GPU memory: 70%
Active requests: 2
Pod B
GPU memory: 92%
Active requests: 8
Pod C
GPU memory: 55%
Active requests: 1
A simple load-balancing mechanism may not understand the difference between these Pods.
Sending the next request to Pod B may create a queue even though Pod C has available capacity.
This becomes more important with large language models because inference requests can be long-running and model servers can maintain in-memory state such as token caches.
Kubernetes Gateway API's Inference Extension was designed specifically around this type of problem. Its endpoint-selection model can use inference-related signals instead of treating every backend Pod as interchangeable.
Start With a Small Cluster
You do not need dozens of GPU nodes to learn the architecture.
A small environment might contain:
1 Kubernetes control plane
2 GPU worker nodes
1 model
2 model server replicas
1 inference gateway
1 client application
For development, the GPU hardware depends on the model you want to run.
The important part is understanding the resource model before choosing hardware.
For example:
resources:
requests:
cpu: "4"
memory: "16Gi"
nvidia.com/gpu: "1"
limits:
cpu: "8"
memory: "32Gi"
nvidia.com/gpu: "1"
The NVIDIA GPU resource is only an example.
Your cluster must have a compatible device plugin or GPU resource implementation installed before Kubernetes can schedule Pods requesting that resource.
The application should never assume that a normal CPU node can execute a GPU workload.
Label the GPU Nodes
If the cluster contains both CPU and GPU nodes, make the scheduling requirement explicit.
For example:
kubectl label nodes gpu-node-1 accelerator=nvidia
kubectl label nodes gpu-node-2 accelerator=nvidia
Then use a node selector:
apiVersion: apps/v1
kind: Deployment
metadata:
name: inference-server
spec:
replicas: 2
selector:
matchLabels:
app: inference-server
template:
metadata:
labels:
app: inference-server
spec:
nodeSelector:
accelerator: nvidia
containers:
- name: model-server
image: your-model-server-image
resources:
limits:
nvidia.com/gpu: "1"
This prevents the inference workload from being scheduled onto an unsuitable node.
For larger environments, node affinity, taints, tolerations, and topology constraints provide more control than a simple node selector.
Keep the Model Server Separate From the Client
A common mistake is putting application logic and model inference into the same Pod.
For example:
Web API
|
+-- Model
|
+-- GPU
That couples two different scaling problems.
Your API might need 10 replicas while the model only needs 2 GPU replicas.
A better design is:
Web API Deployment
|
v
Inference Service
|
v
Model Server Deployment
|
v
GPU Nodes
Now the application tier can scale independently from the expensive inference tier.
This is one of the biggest practical reasons to treat inference as a separate service.
Expose the Model Through a Service
The model servers should normally be hidden behind a Kubernetes Service.
apiVersion: v1
kind: Service
metadata:
name: inference-server
spec:
selector:
app: inference-server
ports:
- name: http
port: 8000
targetPort: 8000
The application can then call:
http://inference-server:8000
instead of knowing the individual Pod IP addresses.
If one Pod is restarted, Kubernetes can update the Service endpoints without requiring the application to discover the new Pod manually.
This is the normal Kubernetes service-discovery model.
The important difference is that the Service should not be considered the complete AI routing layer.
For simple workloads, it may be enough.
For high-throughput or model-aware inference, you may eventually need inference-aware routing.
Add an Inference Gateway
The Gateway API provides a Kubernetes-native model for exposing and routing traffic.
A normal Gateway API flow looks roughly like this:
Client
|
v
Gateway
|
v
HTTPRoute
|
v
Backend Service
|
v
Pods
Gateway API is designed to separate infrastructure responsibilities from application routing configuration. Kubernetes documents Gateway API as the successor to the older Ingress API and describes Gateway and HTTPRoute as separate resources for expressing this routing model.
For AI inference, the Gateway API Inference Extension adds another layer.
The conceptual flow becomes:
Client
|
v
Gateway
|
v
HTTPRoute
|
v
InferenceModel
|
v
InferencePool
|
v
Best model server
The distinction between InferenceModel and InferencePool is useful.
InferenceModel describes what model endpoint an AI application wants.
InferencePool describes the pool of model-serving Pods and how the platform manages them.
This separates the developer's model request from the infrastructure team's compute management.
Understand InferenceModel and InferencePool
Think of the two resources as two different questions.
InferenceModel asks:
Which model should the application call?
InferencePool asks:
Where are the model servers running?
For example:
Application
|
| "chat-model"
v
InferenceModel
|
v
InferencePool
|
+-- Model Pod 1
+-- Model Pod 2
+-- Model Pod 3
This separation becomes valuable when the infrastructure changes.
You could move from two GPU nodes to four GPU nodes without changing the application's model endpoint.
The platform team changes the infrastructure.
The application continues requesting the same logical model.
Model-Aware Routing
Traditional load balancing might use:
Round robin
For example:
Request 1 -> Pod A
Request 2 -> Pod B
Request 3 -> Pod C
Request 4 -> Pod A
Inference-aware routing can consider information such as:
This allows the gateway to choose a backend based on the actual inference workload rather than simply counting Pods.
The Gateway API Inference Extension describes an Endpoint Selection Extension that can use live information such as queue length, memory usage, and loaded adapters when selecting a model server.
That is a significant architectural difference.
Avoid Loading the Model on Every Request
The model should normally remain loaded in the model server's memory.
Do not design the system like this:
Request
|
v
Start container
|
v
Load model
|
v
Run inference
|
v
Delete container
For a large model, model loading can take considerably longer than the actual request processing.
Instead:
Pod starts
|
v
Load model
|
v
Model remains available
|
+---- Request
+---- Request
+---- Request
This is why inference Pods should generally be treated as warm workers.
It also explains why scaling an inference workload is different from scaling a stateless web API.
Readiness Must Include Model Loading
A model server can have a running container while still being unable to serve inference.
For example:
Container: Running
Model: Loading
Inference: Not ready
If Kubernetes considers the Pod ready too early, traffic can reach an uninitialized model server.
Use a readiness probe that reflects actual inference readiness.
For example:
readinessProbe:
httpGet:
path: /health/ready
port: 8000
initialDelaySeconds: 10
periodSeconds: 5
The exact endpoint depends on your model server.
The important point is that readiness should represent the application's ability to accept real inference traffic, not merely whether the process exists.
Configure Startup and Liveness Separately
Startup, readiness, and liveness answer different questions.
Startup asks:
Has the model server finished initializing?
Readiness asks:
Can this Pod receive traffic now?
Liveness asks:
Is the process still functioning?
A large model may need significant startup time.
If liveness is configured too aggressively, Kubernetes can repeatedly restart a model that is simply taking time to load.
That produces a restart loop:
Start
|
Load model
|
Probe fails
|
Restart
|
Load model
|
Probe fails
|
Restart
The Pod never becomes ready.
Use startup probes where appropriate so Kubernetes understands the difference between slow initialization and an unhealthy process.
Control GPU Allocation
GPU resources should be explicitly requested.
For example:
resources:
requests:
nvidia.com/gpu: "1"
limits:
nvidia.com/gpu: "1"
For GPU workloads, the requested resource represents a scheduling constraint.
If your model requires one GPU, do not assume Kubernetes will automatically understand that from CPU and memory requirements.
Also avoid overcommitting GPU capacity.
A node with one physical GPU should not be configured as if it had several independent GPUs unless the underlying GPU partitioning technology and Kubernetes integration actually provide those resources.
Think About GPU Memory Separately
GPU count is not the whole story.
Two model servers may both request:
1 GPU
but have very different memory requirements.
For example:
Model A
1 GPU
12 GB GPU memory
Model B
1 GPU
70 GB GPU memory
Both request one GPU, but the hardware requirements are very different.
Before selecting hardware, measure:
Model weight size
Runtime memory
KV cache requirements
Context length
Batch size
Concurrent requests
Input token size
Output token size
For LLM workloads, context length and concurrency can have a major impact on memory requirements.
Scaling an Inference Deployment
A simple Kubernetes Deployment can use multiple replicas:
spec:
replicas: 2
That gives you:
Inference Pool
Pod A -> GPU
Pod B -> GPU
But replicas alone do not solve every scaling problem.
If every Pod needs a GPU, scaling from two replicas to four requires two additional GPU resources.
The cluster therefore needs enough schedulable accelerator capacity.
This creates a dependency:
Traffic
|
v
Inference replicas
|
v
GPU capacity
|
v
Node capacity
Scaling the Deployment without enough GPU capacity simply leaves Pods Pending.
Use Autoscaling Carefully
AI inference is not always a good candidate for a simple CPU-based Horizontal Pod Autoscaler.
CPU utilization may look normal while GPU memory or inference queue length is becoming the bottleneck.
For example:
CPU: 35%
GPU: 95%
Queue: 24 requests
A CPU-based autoscaler might decide that no scaling is necessary.
An inference-aware scaling system should consider metrics related to the actual bottleneck.
Kubernetes 1.37 also added Beta support for HPA scaling to zero when suitable object or external metrics are available, which can be useful for workloads where idle compute is expensive. However, scale-to-zero introduces cold-start considerations and Kubernetes Services do not buffer HTTP requests while no Pods are ready.
For interactive inference, keeping at least one warm replica may therefore be preferable.
Separate Interactive and Batch Inference
Not every inference request has the same latency requirement.
Consider:
Interactive chat
Target: low latency
Document processing
Target: throughput
Offline embedding
Target: cost efficiency
Putting all three workloads into one inference pool can make capacity planning difficult.
A better architecture may use separate pools:
Inference Gateway
|
+---------------+---------------+
| | |
v v v
Chat Pool Embedding Pool Batch Pool
| | |
GPUs GPUs GPUs
Interactive traffic can receive higher priority.
Batch workloads can use capacity that would otherwise remain idle.
Handle Model Rollouts Carefully
Updating a model is different from deploying a normal web application.
Suppose you replace:
Model v1
with:
Model v2
If all replicas restart together, the entire inference service can become unavailable.
A safer approach is to maintain overlap:
Model v1
|
+-- Pod A
+-- Pod B
Model v2
|
+-- Pod C
+-- Pod D
Traffic can gradually move toward v2.
This is particularly useful when model startup takes several minutes.
The new model should become ready before receiving production traffic.
Keep Model Configuration Outside the Image
Avoid rebuilding the container image for every configuration change.
Use Kubernetes configuration resources for values that are not secrets.
For example:
apiVersion: v1
kind: ConfigMap
metadata:
name: inference-config
data:
MODEL_NAME: "example-model"
MAX_BATCH_SIZE: "8"
MAX_CONTEXT_LENGTH: "8192"
Sensitive values should be handled through an appropriate secret-management mechanism.
The model container should receive configuration through environment variables, mounted files, or another supported configuration mechanism.
This keeps deployment artifacts separate from environment-specific configuration.
Monitor the Right Metrics
CPU and memory are useful, but they are not enough for AI inference.
Track:
Request rate
Request latency
Queue length
Tokens per second
Input tokens
Output tokens
Time to first token
GPU utilization
GPU memory usage
Model load time
Pod restarts
Failed requests
Active requests
For streaming LLM applications, time to first token can be especially useful.
A request may have acceptable total duration while still feeling slow because the first token takes too long.
For batch inference, throughput may be more important than interactive latency.
Metrics should therefore reflect the workload's actual user experience.
Use Kubernetes Events During Troubleshooting
When an inference Pod does not start, begin with:
kubectl get pods -n ai
Then inspect the Pod:
kubectl describe pod <pod-name> -n ai
Look for:
FailedScheduling
Insufficient GPU
ImagePullBackOff
CrashLoopBackOff
Readiness probe failed
If the Pod is Pending because no GPU is available, restarting it will not fix the problem.
The scheduler needs a suitable node.
This distinction saves a lot of time during production incidents.
Check GPU Visibility Inside the Container
A Pod can have a GPU resource request and still have problems at runtime.
Inspect the container:
kubectl exec -it <pod-name> -n ai -- nvidia-smi
If the command is unavailable or the GPU is not visible, investigate:
Do not immediately blame the model server.
The problem may exist several layers below the application.
Design for Node Failure
A small inference cluster should assume that a GPU node can disappear.
For example:
GPU Node A
|
+-- Model Pod A
GPU Node B
|
+-- Model Pod B
If Node A fails, Pod A should be recreated on another suitable node if capacity exists.
But this only works if the cluster has spare GPU capacity.
If both replicas require all available GPUs, Kubernetes cannot recreate the missing replica.
This is why high availability for GPU workloads is partly an infrastructure-capacity problem.
You cannot create two independent inference replicas from one physical GPU unless the underlying hardware and virtualization strategy actually support that arrangement.
Consider Topology for Larger Clusters
As inference workloads grow, physical placement matters.
Two GPUs may communicate more efficiently when they are located within the same server or appropriate hardware topology.
Distributed workloads can also require multiple Pods to be placed in specific locations.
Kubernetes has been adding workload-aware scheduling features specifically for AI and other complex workloads. Kubernetes 1.37 introduced Beta capabilities including gang scheduling, workload-aware preemption, shared ResourceClaims for PodGroups, and more advanced topology-aware scheduling.
These features matter more as the workload moves from a small inference deployment toward distributed or multi-accelerator systems.
Common Mistakes
Treating inference like a normal web application
A GPU model server has different startup, memory, scaling, and routing characteristics.
Scaling without GPU capacity
Increasing replicas does not create physical GPUs.
The additional Pods remain Pending if the cluster cannot satisfy their resource requests.
Using CPU utilization as the only scaling metric
The GPU or inference queue may be saturated while CPU usage remains low.
Sending traffic before model loading finishes
A running container is not necessarily a ready inference server.
Restarting slow model servers
Aggressive liveness probes can restart healthy processes during model initialization.
Loading models for every request
Model loading should normally happen during server initialization, not request processing.
Sending every request through round-robin routing
Round-robin routing may create hotspots when inference Pods have different queue lengths or cached state.
Scaling interactive inference to zero without considering cold starts
Scale-to-zero can save GPU cost but can introduce startup latency.
Treating all inference workloads equally
Chat, embedding, and batch workloads have different latency and throughput requirements.
A Practical Small-Cluster Architecture
A reasonable starting architecture looks like this:
Client
|
v
Gateway / API
|
v
Inference Routing
|
+-------------+-------------+
| |
v v
Chat Model Pool Embedding Pool
| |
+-----+-----+ +-----+-----+
| | | |
v v v v
Model Pod Model Pod Model Pod Model Pod
| | | |
GPU GPU GPU GPU
The surrounding Kubernetes platform handles:
Scheduling
Service discovery
Health checks
Rolling updates
Resource limits
Autoscaling
Secrets
Configuration
Observability
Recovery
The inference layer handles:
Model loading
Token generation
Batching
KV cache
Inference protocol
Model-specific optimization
Keeping those responsibilities separate makes the system easier to operate.
When a Small Kubernetes Inference Cluster Makes Sense
Kubernetes is a good fit when you need several of these capabilities together:
Multiple models
Shared GPU infrastructure
Multiple client applications
Controlled deployments
Centralized observability
Independent model scaling
Standard authentication and networking
Automated recovery
Multi-team access
It may be unnecessary if you only need one model on one machine for local experimentation.
A single inference server can be much simpler.
The operational cost of Kubernetes is justified when the infrastructure itself becomes a shared platform.
Advantages and Disadvantages
Advantages:
Separates applications from model infrastructure.
Provides scheduling and resource management.
Supports multiple model-serving workloads.
Handles failed Pods automatically.
Provides standard networking and service discovery.
Supports controlled model rollouts.
Can scale inference capacity independently from application services.
Provides a foundation for inference-aware routing.
Disadvantages:
GPU infrastructure is expensive.
Model startup times complicate scaling.
GPU scheduling is more complicated than CPU scheduling.
Inference routing can require additional components.
Observability needs are more specialized.
Cluster operations become more complex.
High availability requires spare accelerator capacity.
Kubernetes solves the infrastructure orchestration problem, not every AI serving problem.
The model runtime, GPU hardware, inference protocol, and routing layer still need to be designed correctly.
Production Checklist
Before moving a Kubernetes inference cluster into production, verify:
GPU nodes are correctly configured.
GPU resources are visible to Kubernetes.
Model Pods request the correct accelerator resources.
Model loading happens during startup.
Readiness reflects actual inference availability.
Startup probes account for model initialization time.
Liveness probes do not restart healthy slow-starting processes.
Model servers are exposed through stable Services.
Interactive and batch workloads are separated where necessary.
Model versions can be rolled out gradually.
GPU utilization and memory are monitored.
Queue length and inference latency are monitored.
Time to first token is tracked for streaming workloads.
GPU capacity is sufficient for failure recovery.
Autoscaling uses metrics related to the real inference bottleneck.
Scale-to-zero is used only where cold starts are acceptable.
Logs do not expose prompts or sensitive model inputs unnecessarily.
Gateway routing is tested under concurrent load.
Model upgrades are tested before production rollout.
Summary
A small AI inference cluster is more than a collection of GPU Pods.
The useful architecture separates the application, routing, model-serving, and infrastructure layers.
The application calls a logical model endpoint.
The inference layer decides which model server should process the request.
Kubernetes schedules and manages the underlying Pods and GPU resources.
For simple workloads, a Kubernetes Service may be enough. As inference traffic becomes more complex, inference-aware routing becomes increasingly useful because model servers are not always interchangeable. The Gateway API Inference Extension is designed around this problem by introducing concepts such as InferenceModel, InferencePool, and endpoint selection based on inference workload information.
The most important design decision is to avoid treating AI inference as ordinary stateless HTTP traffic.
Models take time to load.
GPU memory is limited.
Requests can remain active for a long time.
Different models have different resource requirements.
Scaling requires actual accelerator capacity.
Once these differences are built into the architecture, Kubernetes becomes a useful foundation for running a small shared inference platform and gradually expanding it as model traffic grows.
Join the conversation! Your thoughts help the community grow.