Running Kubernetes AI Workloads Across Multiple GPU Nodes Without Wasting Capacity

A Pending AI Pod is not necessarily an application failure.

It may mean:

No GPU available

or:

GPU type mismatch

or:

Insufficient CPU

or:

Insufficient memory

or:

Topology requirements cannot be satisfied

Start troubleshooting with:

kubectl get pods -n ai

Then:

kubectl describe pod <pod-name> -n ai

Look at the scheduling events.

For example:

0/8 nodes are available:
2 insufficient nvidia.com/gpu
3 node affinity conflict
2 taint not tolerated
1 insufficient memory

That message is far more useful than repeatedly restarting the Pod.

Check Node Capacity

Inspect the nodes:

kubectl get nodes

Then:

kubectl describe node <node-name>

Look for:

Capacity
Allocatable
Allocated resources

The difference matters.

A node may physically contain a GPU, but Kubernetes may not advertise it as an allocatable resource.

In that situation, the problem is not scheduling policy.

The accelerator integration needs investigation.

Check GPU Device Discovery

For NVIDIA-based environments, inspect the node and GPU visibility.

A basic check might be:

nvidia-smi

From the Kubernetes side, inspect the advertised resources:

kubectl describe node <node-name>

Look for a resource such as:

nvidia.com/gpu: 4

If the node has physical GPUs but Kubernetes reports zero, investigate:

• GPU drivers

• Device plugin

• Container runtime

• Node configuration

• Version compatibility

The scheduler can only schedule resources that the cluster exposes correctly.

Understand GPU Sharing

Some environments support sharing a physical GPU between workloads.

There are different approaches, including:

• Time-slicing

• Hardware partitioning

• Virtualization

• MIG on supported NVIDIA hardware

These approaches have different isolation and performance characteristics.

For example, time-sharing may allow several workloads to use one physical GPU, but the workloads still compete for the same underlying compute and memory resources.

Hardware partitioning can provide stronger boundaries on supported devices.

Do not assume that requesting:

0.25 GPU

automatically gives a Pod one-quarter of a GPU.

The cluster's accelerator integration determines how sharing works.

Use GPU Partitioning for Smaller Models

Suppose a small embedding model uses only part of a GPU.

Running one model per physical GPU could leave capacity unused.

If the hardware and platform support GPU partitioning, several workloads may be placed on one accelerator.

Conceptually:

Physical GPU
 |
 +-- Partition A -> Embedding model
 |
 +-- Partition B -> Reranker
 |
 +-- Partition C -> Small inference model

This can improve utilization.

But partitioning also introduces limits.

A workload that later needs more memory cannot necessarily borrow memory from another partition.

Choose partition sizes based on measured workloads.

Avoid Packing by GPU Count Alone

Two workloads may both request:

1 GPU

but have very different memory and throughput requirements.

For example:

Model A
1 GPU
10 GB memory

Model B
1 GPU
70 GB memory

If both are treated as identical scheduling units, placement can be inefficient.

Use hardware labels, node pools, partitioning, or other mechanisms when the distinction matters.

The Kubernetes scheduler cannot infer application-level model requirements from the number one.

Think About Network Bandwidth

Distributed AI workloads can generate large amounts of network traffic.

For example:

GPU 1
  |
  +---- Network ---- GPU 2
  |
  +---- Network ---- GPU 3
  |
  +---- Network ---- GPU 4

If the workload frequently exchanges large tensors, network bandwidth and latency can become important.

A cluster with enough GPUs but insufficient network performance may still deliver poor results.

For multi-node training, consider:

• Network bandwidth

• Latency

• RDMA support where required

• Topology

• Placement

• Storage throughput

Do not evaluate an AI cluster based on GPU count alone.

Storage Can Become Another Bottleneck

Model weights can be large.

If every new inference Pod downloads several gigabytes before becoming ready, scaling can be slow.

For example:

Pod starts
   |
   v
Download model
   |
   v
Load model
   |
   v
Initialize runtime
   |
   v
Ready

A sudden scale-out can therefore create significant storage and network traffic.

Use appropriate image and model distribution strategies.

Where supported, consider:

• Local model caches

• Shared high-performance storage

• Preloaded model artifacts

• Efficient container images

• Model repositories

The goal is to reduce the time between Pod scheduling and actual inference readiness.

Separate Model Storage From Container Images

A model does not always need to be embedded directly inside the container image.

For example:

Container image
    |
    +-- Model server
    +-- Runtime
    +-- Dependencies

Model storage
    |
    +-- Model weights
    +-- Tokenizer
    +-- Configuration

This allows the model to change independently from the serving image.

However, external model storage adds its own startup dependency.

If the storage system is slow or unavailable, the model server may never become ready.

Choose the approach based on operational requirements.

Design for GPU Node Failure

A production inference cluster should assume that a GPU node can fail.

Suppose:

Node A
4 GPUs

Node B
4 GPUs

If Node A fails, the cluster loses half its inference capacity.

Kubernetes can recreate the Pods, but only if Node B or another available node has enough capacity.

This creates a useful distinction:

Pod redundancy

versus:

Hardware redundancy

Three replicas on one GPU node do not provide the same failure protection as replicas spread across multiple GPU nodes.

Use topology spread constraints when appropriate.

Spread Replicas Across Nodes

A workload with three replicas should ideally avoid placing all three on one node when availability matters.

For example:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: inference-server

This tells Kubernetes to distribute matching Pods across the specified topology domain.

The exact policy should reflect the workload.

For a development environment, strict spreading may waste capacity.

For production inference, it can provide better fault isolation.

Do Not Over-Spread Tiny Clusters

There is a tradeoff.

Suppose the cluster contains only two GPU nodes and a workload requires four GPUs.

A strict spreading rule may make scheduling impossible.

Topology constraints should therefore be evaluated alongside:

• GPU count

• Node count

• Model requirements

• Availability goals

• Cost

Scheduling rules are not free.

Every additional constraint reduces the scheduler's placement options.

Use a Dedicated Inference Namespace

A dedicated namespace provides a useful operational boundary.

For example:

ai-inference
ai-training
ai-batch

The namespace can contain:

Deployments
Services
ConfigMaps
Secrets
HPA
ResourceQuotas
NetworkPolicies

This makes it easier to apply policies consistently.

It also makes resource usage easier to monitor by workload category.

Add Network Policies

AI inference services may handle sensitive prompts or generated responses.

Do not allow every Pod in the cluster to communicate freely with the model servers.

A simple policy can restrict access to approved application namespaces.

For example:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: inference-access
  namespace: ai-inference
spec:
  podSelector:
    matchLabels:
      app: inference-server
  policyTypes:
    - Ingress

The exact rules depend on the cluster networking implementation.

The important design principle is to treat inference endpoints as internal services that should have controlled access.

Monitor Utilization at Multiple Levels

A useful AI capacity dashboard should show:

Cluster
 |
 +-- GPU capacity
 +-- GPU allocated
 +-- GPU free
 |
 +-- Pending Pods
 |
 +-- Node health
 |
 +-- Inference latency
 +-- Queue length
 +-- Throughput
 |
 +-- Training jobs
 +-- Batch jobs

You need both infrastructure and application metrics.

For example:

GPU utilization: 40%
Pending Pods: 8

This could indicate that the GPUs are the wrong type for the pending workloads.

Alternatively:

GPU utilization: 95%
Pending Pods: 0

This may indicate good utilization.

The numbers only become useful when you understand the workload behind them.

Build Capacity Around Peak Traffic

Do not size the cluster only for average traffic.

Suppose:

Average:
2 inference requests/second

Peak:
20 requests/second

The cluster must handle the peak according to the required latency target.

However, buying enough GPUs to permanently handle the peak may be expensive.

Use autoscaling, batch scheduling, priority, and workload separation to manage predictable peaks.

For workloads with strict latency requirements, maintaining warm capacity may be necessary.

For batch workloads, queueing can be acceptable.

A Practical Multi-GPU Cluster Architecture

A production-oriented cluster might look like this:

                         AI Gateway
                              |
                 +------------+------------+
                 |                         |
                 v                         v
          Interactive Pool            Batch Pool
                 |                         |
          +------+------+             +----+----+
          |             |             |         |
          v             v             v         v
       Model A       Model B       Job 1      Job 2
          |             |             |         |
       GPU Node       GPU Node     GPU Node  GPU Node
          |             |             |         |
          +-------------+-------------+---------+
                        |
                        v
                Kubernetes Scheduler
                        |
          +-------------+-------------+
          |                           |
          v                           v
   Hardware Constraints        Resource Policies
   - GPU type                  - Priority
   - GPU memory                - Quota
   - Topology                  - Affinity
   - Network                   - Taints

This is more realistic than treating the entire GPU fleet as one undifferentiated resource pool.

Common Mistakes

Counting GPUs without checking their type

Eight GPUs are not necessarily equivalent to eight interchangeable GPUs.

Allowing ordinary workloads onto GPU nodes

This can waste expensive accelerator capacity and complicate scheduling.

Requesting a GPU without CPU or memory requirements

The Pod may still remain Pending because another resource is exhausted.

Treating multi-GPU jobs as independent Pods

Some distributed workloads require coordinated scheduling and topology.

Using priority for everything

If every workload is high priority, priority loses its purpose.

Assuming GPU sharing is automatic

GPU sharing depends on the accelerator platform and its configuration.

Ignoring model download time

A scale-out event can take much longer when every Pod needs to download large model artifacts.

Placing all replicas on one node

This creates a single hardware failure domain.

Adding too many scheduling constraints

Over-constrained workloads may become impossible to schedule even when the cluster has free resources.

Measuring only GPU utilization

Pending Pods, queue length, memory usage, and latency provide important context.

Production Checklist

Before running multiple AI workloads on a shared Kubernetes GPU cluster, verify:

• GPU nodes are correctly registered with Kubernetes.

• GPU types are labeled where hardware differences matter.

• GPU memory capacity is known.

• GPU nodes are protected with taints when appropriate.

• AI workloads have explicit GPU requests.

• CPU and memory requirements are also defined.

• Interactive and batch workloads are separated when necessary.

• Training workloads have appropriate scheduling rules.

• Priority classes represent real business importance.

• Resource quotas prevent one team from consuming the entire cluster.

• Autoscaling has sensible maximum limits.

• Multi-GPU jobs have appropriate topology requirements.

• Gang scheduling is used only where coordinated startup is necessary.

• Replicas are spread across failure domains where availability matters.

• GPU sharing is configured deliberately.

• Model startup and download time are measured.

• Model storage is reliable and fast enough for scaling events.

• Network bandwidth is sufficient for distributed workloads.

• Pending Pods are monitored.

• GPU, CPU, memory, and queue utilization are visible.

• Inference and training traffic do not unintentionally compete for the same capacity.

• Node failure scenarios have been tested.

Summary

A Kubernetes AI cluster should not be designed around GPU count alone.

The real scheduling problem is:

Workload requirements
        +
Hardware capabilities
        +
Topology
        +
Priority
        +
Capacity

A one-GPU inference workload is relatively simple.

A shared cluster containing different models, batch jobs, training workloads, and multi-GPU applications requires more careful scheduling.

Use labels and affinity to select appropriate hardware.

Use taints and tolerations to keep unrelated workloads away from expensive GPU nodes.

Use resource requests and quotas to control capacity.

Use priority for workloads that genuinely need preferential scheduling.

For distributed workloads, consider topology and gang scheduling so that the scheduler understands that several resources may need to become available together. Recent Kubernetes releases continue to add workload-aware scheduling features for AI and other resource-intensive applications.

Most importantly, monitor the entire path.

A GPU can be idle while Pods are Pending because the available GPUs are the wrong type.

A GPU can be busy while latency remains acceptable.

A cluster can have enough total GPUs but still fail to schedule a four-GPU job because those GPUs are fragmented across nodes.

Good AI infrastructure therefore comes from understanding how the workload actually uses hardware rather than simply adding more accelerators.

When Kubernetes scheduling, GPU topology, inference routing, quotas, and workload priorities are designed together, a shared GPU cluster can support more AI workloads without turning every scaling problem into another expensive hardware purchase.