Kubernetes 1.37 Gang Scheduling: Improving Distributed AI Workloads

Distributed AI workloads are different from many traditional applications running in Kubernetes.

A web application may continue working when Kubernetes starts a few additional Pods later. A distributed training workload often needs several Pods to become available together. If only part of the workload is scheduled, the remaining Pods may sit idle while the running Pods consume resources without making useful progress.

This is where gang scheduling becomes important.

Gang scheduling is a scheduling approach where a group of related Pods is considered together. Instead of scheduling individual Pods independently, the scheduler attempts to place the required group when the necessary resources are available.

For AI and machine-learning workloads running on Kubernetes, this can improve resource utilization and reduce situations where partially scheduled distributed jobs occupy cluster capacity.

What Is Gang Scheduling?

Imagine a distributed training job requiring four Pods:

AI Training Job
      |
      +-- Worker 1
      +-- Worker 2
      +-- Worker 3
      +-- Worker 4

With ordinary scheduling, Kubernetes may schedule:

Worker 1 → Running
Worker 2 → Running
Worker 3 → Pending
Worker 4 → Pending

The first two workers consume resources while the distributed job cannot start properly.

With gang scheduling, the scheduler considers the group:

Worker 1
Worker 2
Worker 3
Worker 4
   |
   v
Can the required group be scheduled?
      |
   +-- Yes → Schedule group
   |
   +-- No  → Keep group waiting

The exact behavior depends on the scheduling implementation and workload configuration.

Why Distributed AI Workloads Need It

Distributed training commonly divides work across multiple workers.

A simplified architecture is:

                Training Job
                     |
       +-------------+-------------+
       |             |             |
       v             v             v
   Worker 1      Worker 2      Worker 3
       |             |             |
       +-------------+-------------+
                     |
                  Dataset

The workers may need to communicate with one another before meaningful computation can proceed.

If resources are fragmented across the cluster, Kubernetes might schedule only some of the workers.

That creates:

Allocated resources
        ↓
Partial workload
        ↓
Waiting workers
        ↓
Poor utilization

Gang scheduling addresses this class of scheduling problem by considering workload groups rather than treating every Pod as completely independent.

A Simple Kubernetes Workload

Consider a distributed workload with four replicas:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: training-workers
spec:
  replicas: 4
  selector:
    matchLabels:
      app: training-worker
  template:
    metadata:
      labels:
        app: training-worker
    spec:
      containers:
        - name: worker
          image: example/training-worker:1.0
          resources:
            requests:
              cpu: "2"
              memory: "4Gi"

This declares four Pods, but a Deployment by itself does not express the requirement:

"Schedule all four together."

That distinction matters.

A distributed application may require coordination that ordinary replica management does not provide.

Gang Scheduling and Resource Requests

Resource requests are important because the scheduler uses them when determining whether Pods can fit on nodes.

For example:

resources:
  requests:
    cpu: "4"
    memory: "8Gi"

If a distributed job requires eight workers:

8 × 4 CPU = 32 CPU
8 × 8 GiB = 64 GiB

the cluster needs sufficient allocatable resources to schedule the workload.

Gang scheduling does not create additional capacity.

It changes how the scheduler handles a workload that requires multiple resources at the same time.

The Resource Fragmentation Problem

Consider a cluster:

Node A → 4 CPU available
Node B → 4 CPU available
Node C → 4 CPU available
Node D → 4 CPU available

A distributed workload requires:

4 Pods × 4 CPU

In total, the cluster has enough CPU.

But if the scheduling requirements, affinity rules, topology constraints, or other workloads prevent the Pods from being placed appropriately, the job may remain partially scheduled.

The distinction is:

Total Capacity
      ≠
Immediately Usable Capacity

Gang scheduling is useful when workload coordination matters in addition to raw resource availability.

Kubernetes Scheduling Concepts

Kubernetes scheduling involves several concepts that influence where Pods run:

Node resources
Affinity
Anti-affinity
Taints
Tolerations
Topology constraints
Priority
Preemption
Scheduling policies

Gang scheduling needs to work alongside these mechanisms.

For example, an AI workload may require workers to be distributed across nodes.

A configuration can use topology constraints such as:

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app: training-worker

This controls how Pods are distributed across topology domains.

It does not by itself turn the workload into a gang-scheduled workload.

Gang Scheduling vs Pod Scheduling

The conceptual difference is:

Traditional Scheduling	Gang Scheduling
Considers Pods individually	Considers a group
Partial placement can occur	Group placement can be coordinated
Useful for independent workloads	Useful for tightly coupled workloads
Resource allocation is per Pod	Resource requirements can be considered collectively
May cause partially running distributed jobs	Can reduce partial allocation

Gang scheduling is therefore not intended to replace normal Kubernetes scheduling.

It addresses a particular class of workloads.

Why AI Training Is a Good Example

Distributed AI training often contains:

Coordinator
     |
     +-- Worker 1
     +-- Worker 2
     +-- Worker 3
     +-- Worker 4

The workers may need:

GPU
CPU
Memory
Network
Storage

at the same time.

Suppose a job requires four GPUs but the cluster currently has only three usable GPUs.

Without appropriate group scheduling:

Worker 1 → GPU
Worker 2 → GPU
Worker 3 → GPU
Worker 4 → Pending

The first three Pods have already consumed resources.

A gang-oriented approach can instead keep the workload waiting until the required group can be scheduled, depending on the scheduler and workload implementation.

GPU Workloads

AI training often uses specialized hardware.

A Pod might request:

resources:
  limits:
    nvidia.com/gpu: 1

If a workload requires eight workers:

8 workers
×
1 GPU
=
8 GPUs

The cluster needs enough compatible GPUs.

Other constraints can also matter:

GPU model
Node availability
Topology
Interconnect
Memory
Taints

Gang scheduling is useful for coordinating the workload, but it does not solve hardware shortages.

Gang Scheduling and Batch Jobs

Distributed AI workloads are frequently batch-oriented.

For example:

Training Job
   ↓
Acquire Resources
   ↓
Start Workers
   ↓
Train
   ↓
Save Model
   ↓
Complete

A partially scheduled job can create unnecessary cluster contention.

Gang scheduling can be particularly useful when the job has a minimum number of workers required to make progress.

For example:

Minimum workers = 8
Requested workers = 8

The scheduler should not treat:

3 workers

as an acceptable partial execution if the application cannot perform useful work with only three workers.

Minimum Available Resources

A distributed workload can conceptually define:

minimum resources required

For example:

Minimum workers: 4
Desired workers: 8

The distinction can be useful for workloads that support elastic scaling.

An application that supports elasticity may be able to operate with fewer workers.

A tightly synchronized training job may not.

The scheduling policy should reflect the application's actual execution model.

Gang Scheduling and Preemption

Consider a cluster where resources are occupied:

Existing workloads
      ↓
Available capacity = insufficient
      ↓
AI training job waits

If priority and preemption are configured, Kubernetes can potentially remove lower-priority Pods to make room for higher-priority workloads.

For example:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: distributed-training
value: 100000
globalDefault: false
description: "Priority for distributed training workloads"

The exact priority values should reflect the organization's scheduling policy.

High priority should not be assigned casually.

Otherwise, AI workloads can starve other important services.

Gang Scheduling Is Not the Same as Priority

These mechanisms solve different problems.

Priority
   ↓
Which workload should win?

Gang Scheduling
   ↓
How should a related group be scheduled?

They can complement each other.

For example:

Priority
   +
Gang Scheduling
   +
Resource Requests
   +
Topology Constraints

can provide a more complete scheduling strategy for distributed workloads.

Kubernetes 1.37 Considerations

When adopting Kubernetes 1.37 scheduling capabilities, teams should verify which gang-scheduling functionality is available in their exact Kubernetes distribution and configuration.

Kubernetes scheduling features can depend on:

Kubernetes version
Feature gates
Scheduler configuration
Scheduler plugins
Workload API
Cloud distribution

Do not assume that every Kubernetes 1.37 cluster has identical scheduling behavior.

The scheduler configuration should be treated as part of the platform configuration and tested before production rollout.

Testing a Distributed Workload

A useful test starts with a deliberately constrained cluster.

For example:

Cluster capacity:
4 GPUs

Job requirement:
4 GPUs

Then submit the distributed workload.

Observe:

kubectl get pods -o wide

and:

kubectl describe pod <pod-name>

Look at scheduling events and determine whether the workload is being scheduled as intended.

Monitoring Pending Pods

A distributed job can become stuck because resources are unavailable.

Use:

kubectl get pods

A typical result might show:

training-worker-1   Pending
training-worker-2   Pending
training-worker-3   Running
training-worker-4   Pending

The next step is to inspect the pending Pod:

kubectl describe pod training-worker-1

Scheduling events can reveal whether the problem is:

Insufficient CPU
Insufficient memory
Insufficient GPU
Taint
Affinity rule
Topology constraint
Preemption

Testing Resource Fragmentation

Do not test only a completely empty cluster.

A realistic test should include other workloads.

For example:

Node 1 → Partially occupied
Node 2 → Partially occupied
Node 3 → Partially occupied
Node 4 → Partially occupied

Then submit the distributed job.

This can reveal behavior that is invisible when the cluster has abundant unused capacity.

Testing Job Cancellation

Distributed workloads should also be tested when they are cancelled.

For example:

Training Job
     ↓
8 workers
     ↓
Cancellation
     ↓
Workers terminate
     ↓
Resources become available

Check that resources are released properly.

A failed or cancelled distributed workload should not leave unnecessary Pods consuming capacity.

Testing Node Failure

Distributed AI workloads should also be tested against node failures.

For example:

Worker 1 → Node A
Worker 2 → Node B
Worker 3 → Node C
Worker 4 → Node D
                 X
              Node D fails

The workload's behavior depends on the application and scheduler configuration.

Important questions include:

Can the worker be rescheduled?
Does the training job restart?
Does the application recover?
Are partially running workers useful?
Is checkpointing available?

Gang scheduling helps with scheduling coordination, but it does not automatically make the distributed application fault tolerant.

Common Mistakes

Assuming Gang Scheduling Creates Capacity

It does not add CPU, memory, or GPUs to a cluster.

Using It for Every Deployment

Independent web applications generally do not require gang scheduling.

Ignoring Resource Requests

The scheduler needs accurate resource requirements.

Ignoring GPU Availability

A distributed AI job cannot schedule GPUs that do not exist or are already allocated.

Confusing Priority With Gang Scheduling

Priority determines scheduling preference; gang scheduling addresses coordinated group placement.

Testing Only an Empty Cluster

Real scheduling problems often appear when resources are fragmented.

Ignoring Application Recovery

Scheduling coordination does not replace distributed application fault tolerance.

Troubleshooting

Pods Remain Pending

Start with:

kubectl describe pod <pod-name>

Review scheduling events.

Insufficient CPU or Memory

Compare Pod requests with node allocatable resources.

Insufficient GPU

Check:

kubectl describe nodes

and verify that GPU resources are advertised correctly.

Affinity Prevents Scheduling

Review:

nodeAffinity:

and:

podAffinity:

Taints Prevent Placement

Check node taints:

kubectl describe node <node-name>

Then determine whether the workload requires a matching toleration.

Distributed Job Starts Partially

Verify whether the workload is configured with the intended minimum group size and whether the selected scheduling mechanism actually enforces that requirement.

Best Practices

Use gang scheduling for workloads that genuinely require coordinated Pod placement.
Define accurate CPU, memory, and GPU requests.
Test scheduling with realistic cluster utilization.
Combine scheduling coordination with appropriate topology constraints.
Use priorities carefully.
Monitor Pending Pods and scheduler events.
Test node failures separately from scheduling behavior.
Validate the exact Kubernetes distribution and scheduler configuration.
Keep distributed workloads isolated from unnecessary production resource contention.
Ensure cancelled jobs release their resources correctly.

Advantages and Disadvantages

Advantages

Helps coordinate scheduling of tightly coupled workloads.
Can reduce partially scheduled distributed jobs.
Useful for distributed AI and batch workloads.
Can improve resource utilization for workloads requiring multiple Pods together.
Works alongside other Kubernetes scheduling mechanisms.

Disadvantages

Adds scheduling complexity.
Does not create additional cluster capacity.
Requires careful workload and scheduler configuration.
May increase waiting time when sufficient resources are unavailable.
Not necessary for ordinary independent applications.
Does not solve application-level fault tolerance.

Conclusion

Gang scheduling addresses a specific problem in Kubernetes: some workloads are meaningful only when a group of Pods can run together.

That makes it particularly relevant to distributed AI workloads:

Distributed Job
      ↓
Required Worker Group
      ↓
Check Cluster Capacity
      ↓
Group Can Run?
   /          \
 Yes           No
  |             |
Schedule       Wait
  |
  v
Start Training

The important distinction is that gang scheduling is not simply a performance optimization. It is a scheduling strategy for workloads with coordinated resource requirements.

For AI platforms, accurate resource requests, GPU availability, topology constraints, priority policies, and application-level fault tolerance remain equally important.

Teams evaluating Kubernetes 1.37 should verify the exact gang-scheduling capabilities available in their environment and test them under realistic resource pressure. The strongest validation is not an empty-cluster demo, but a controlled experiment where competing workloads, limited resources, and distributed worker requirements are present at the same time.