Architecting an AI-Driven Kubernetes Auto-Scaling Control Plane

Traditional Kubernetes Horizontal Pod Autoscaling (HPA) is inherently reactive, relying on rigid thresholds that often lead to resource underutilization or costly over-provisioning—especially for expensive GPU workloads. In this end-to-end guide, we explore how to implement robust HPA for CPU and GPU workloads, define the critical custom metrics required, and ultimately build an Enterprise Multi-Agent LangGraph RAG system with persistent memory and state to intelligently orchestrate, analyze, and automate these scaling decisions in real-time.

Part 1: Implementing HPA for GPU/CPU Workloads

To scale workloads effectively, Kubernetes requires visibility into resource consumption. While CPU and Memory are native, GPU and application-level metrics require a custom pipeline.

1. The Metrics Pipeline

CPU/Memory: Handled natively by the Kubernetes Metrics Server.
GPU Metrics: Requires the NVIDIA GPU Operator and DCGM (Data Center GPU Manager) Exporter. DCGM exposes hardware-level telemetry (utilization, memory, temperature) to Prometheus.
The Bridge: The Prometheus Adapter scrapes Prometheus and translates these metrics into the Kubernetes Custom Metrics API, making them available to the HPA controller.

2. What Custom Metrics Do We Use?

Relying solely on GPU utilization (DCGM_FI_DEV_GPU_UTIL) is a trap; it can spike to 100% while the application is actually bottlenecked by memory or I/O. For enterprise AI/ML workloads, we use a composite of metrics.

Hardware Metrics (via DCGM)

DCGM_FI_DEV_GPU_UTIL: GPU compute utilization.
DCGM_FI_DEV_FB_USED_PERCENT: VRAM utilization (critical for preventing OOM errors in LLM inference).

Application/Queue Metrics (via Prometheus/OTel)

inference_queue_depth: Number of pending requests in the model server (e.g., vLLM, Triton). This is the ultimate leading indicator for scaling.
p99_inference_latency: 99th percentile request latency.
tokens_per_second_per_gpu: Throughput efficiency.

Part 2: Real-Time Use Case — Enterprise LLM Inference Platform

The Scenario

An enterprise runs a multi-tenant LLM inference service on Kubernetes using vLLM. Traffic is highly spiky.

The Problem

A standard HPA configured for 80% GPU utilization reacts too slowly to sudden queue spikes, causing SLA breaches. Conversely, during low traffic, it keeps too many pods alive, burning cloud compute budgets.

The Solution

We deploy an AI Control Plane powered by LangGraph. Instead of static HPA rules, a Multi-Agent RAG system continuously analyzes real-time telemetry, retrieves historical scaling post-mortems and documentation (RAG), maintains state across scaling cycles (Memory), and dynamically patches the HPA configurations or triggers node-autoscaling.

Part 3: The LangGraph Multi-Agent RAG Architecture

To build this intelligent control plane, we use LangGraph, which allows us to define cyclical, stateful agent workflows.

The Agents

Telemetry Agent

Queries the Prometheus/K8s API for real-time GPU, CPU, and Queue metrics.

RAG Knowledge Agent

Retrieves context from the enterprise Vector Database (e.g., past scaling incidents, vLLM tuning guides, FinOps budgets).

Decision & Actuation Agent

Synthesizes the telemetry and RAG context, updates the graph state, and generates the K8s HPA patch.

State and Memory

State

A TypedDict tracking current metrics, retrieved documents, and the proposed action.

Memory

We use a persistent Checkpointer (e.g., SQLite/Postgres) so the agents remember past scaling decisions. This prevents "scaling oscillation" (rapidly scaling up and down) by allowing the agent to say, "We scaled up 3 minutes ago, let's wait and observe."

Part 4: Code Implementation

Below is the enterprise-grade Python implementation using langgraph, langchain, and the Kubernetes client.

1. Prerequisites & Imports

import operator
from typing import Annotated, Sequence, TypedDict, Literal
from langchain_core.messages import BaseMessage, HumanMessage, AIMessage, SystemMessage
from langchain_openai import ChatOpenAI, OpenAIEmbeddings
from langchain_core.prompts import ChatPromptTemplate
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.sqlite import SqliteSaver
from langchain_community.vectorstores import Chroma
from kubernetes import client, config
import json

# Initialize K8s client (assumes running inside cluster or kubeconfig is loaded)
config.load_incluster_config()
k8s_apps_v1 = client.AppsV1Api()
k8s_autoscaling_v1 = client.AutoscalingV1Api()

2. Defining the State and Memory

We define a strict state schema. The scaling_history acts as our short-term memory within the graph, while the SqliteSaver provides long-term persistent memory across graph invocations.

class AgentState(TypedDict):
    # The current messages in the agent loop
    messages: Annotated[Sequence[BaseMessage], operator.add]

    # Real-time telemetry data
    current_metrics: dict

    # Context retrieved from RAG
    rag_context: str

    # Proposed K8s HPA modifications
    proposed_hpa_patch: dict

    # Memory: History of recent actions to prevent oscillation
    scaling_history: list

    # Final status
    next_action: Literal["scale_up", "scale_down", "hold", "finish"]

3. Building the Agent Nodes

Node A: Telemetry Agent

Fetches real-time data from Prometheus and K8s.

def telemetry_node(state: AgentState):
    # Mocking Prometheus API call for DCGM and Queue metrics
    metrics = {
        "gpu_utilization": 92.5,
        "vram_used_percent": 88.0,
        "inference_queue_depth": 450,
        "p99_latency_ms": 1200
    }

    # Fetch current HPA state
    hpa = k8s_autoscaling_v1.read_namespaced_horizontal_pod_autoscaler(
        name="vllm-inference-hpa",
        namespace="ai-platform"
    )

    metrics["current_replicas"] = hpa.status.current_replicas
    metrics["min_replicas"] = hpa.spec.min_replicas
    metrics["max_replicas"] = hpa.spec.max_replicas

    return {
        "current_metrics": metrics,
        "messages": [AIMessage(content="Telemetry collected.")]
    }

Node B: RAG Knowledge Agent

Queries the Vector DB for enterprise context based on the current metrics.

# Initialize Vector DB (Chroma) with enterprise docs
embeddings = OpenAIEmbeddings()
vectorstore = Chroma(
    collection_name="k8s_finops_and_ops",
    embedding_function=embeddings
)

def rag_node(state: AgentState):
    metrics = state["current_metrics"]

    # Formulate query based on anomalies
    query = (
        f"GPU utilization is {metrics['gpu_utilization']}%, "
        f"Queue depth is {metrics['inference_queue_depth']}. "
    )
    query += (
        "What are the best practices for scaling vLLM under these conditions? "
        "Check past incident reports."
    )

    docs = vectorstore.similarity_search(query, k=3)

    context = "\n---\n".join(
        [doc.page_content for doc in docs]
    )

    return {
        "rag_context": context,
        "messages": [AIMessage(content="RAG context retrieved.")]
    }

Node C: Decision & Actuation Agent (The Orchestrator)

Uses an LLM to decide the next step, utilizing the state and memory.

llm = ChatOpenAI(model="gpt-4o", temperature=0)

def decision_node(state: AgentState):
    prompt = ChatPromptTemplate.from_template("""
    You are an expert Kubernetes FinOps and MLOps AI Agent.

    CURRENT METRICS: {metrics}
    RETRIEVED KNOWLEDGE (RAG): {context}
    RECENT SCALING HISTORY (MEMORY): {history}

    Analyze the situation. If the queue depth is high and GPU is maxed, we need to scale up.
    If queue is empty and GPU is idle, scale down.

    IMPORTANT: Check the scaling_history. If we scaled in the last 3 minutes,
    choose 'hold' to prevent oscillation.

    Respond ONLY in JSON format:
    {{
        "reasoning": "Your thought process",
        "next_action": "scale_up" | "scale_down" | "hold",
        "hpa_patch": {{
            "spec": {{
                "minReplicas": X,
                "maxReplicas": Y
            }}
        }}
    }}
    """)

    chain = prompt | llm

    response = chain.invoke({
        "metrics": json.dumps(state["current_metrics"]),
        "context": state["rag_context"],
        "history": json.dumps(state["scaling_history"][-3:])
    })

    decision = json.loads(response.content)

    # Update history for memory
    history = state["scaling_history"] + [{
        "action": decision["next_action"],
        "metrics": state["current_metrics"]
    }]

    return {
        "proposed_hpa_patch": decision.get("hpa_patch", {}),
        "next_action": decision["next_action"],
        "scaling_history": history,
        "messages": [
            AIMessage(content=f"Decision made: {decision['next_action']}")
        ]
    }

Node D: Actuation Node

Applies the patch to the Kubernetes cluster if a scale event is triggered.

def actuation_node(state: AgentState):
    if state["next_action"] in ["scale_up", "scale_down"]:
        patch = state["proposed_hpa_patch"]

        try:
            k8s_autoscaling_v1.patch_namespaced_horizontal_pod_autoscaler(
                name="vllm-inference-hpa",
                namespace="ai-platform",
                body=patch
            )

            msg = (
                f"Successfully patched HPA. "
                f"Action: {state['next_action']}"
            )

        except Exception as e:
            msg = f"Failed to patch HPA: {str(e)}"

    else:
        msg = "Holding current state. No patch applied."

    return {
        "messages": [AIMessage(content=msg)],
        "next_action": "finish"
    }

4. Compiling the LangGraph with Persistent Memory

Here we wire the nodes together, define the conditional edges, and attach the SqliteSaver for persistent memory.

import sqlite3

def route_decision(state: AgentState):
    if state["next_action"] == "finish":
        return "end"
    return "actuate"

# Initialize persistent memory
# Enterprise grade: use Postgres in production
conn = sqlite3.connect(
    "agent_memory.db",
    check_same_thread=False
)

memory = SqliteSaver(conn)

# Build the Graph
workflow = StateGraph(AgentState)

workflow.add_node("telemetry", telemetry_node)
workflow.add_node("rag", rag_node)
workflow.add_node("decision", decision_node)
workflow.add_node("actuate", actuation_node)

workflow.set_entry_point("telemetry")

workflow.add_edge("telemetry", "rag")
workflow.add_edge("rag", "decision")

workflow.add_conditional_edges(
    "decision",
    route_decision,
    {
        "actuate": "actuate",
        "end": END
    }
)

workflow.add_edge("actuate", END)

# Compile with Checkpointer for Memory
app = workflow.compile(checkpointer=memory)

5. Execution

We run the graph using a thread_id to isolate the memory state for this specific cluster/namespace context.

if __name__ == "__main__":
    config = {
        "configurable": {
            "thread_id": "prod-ai-platform-ns-1"
        }
    }

    initial_state = {
        "messages": [
            HumanMessage(
                content="Initiate scaling evaluation cycle."
            )
        ],
        "current_metrics": {},
        "rag_context": "",
        "proposed_hpa_patch": {},
        "scaling_history": [],
        "next_action": "hold"
    }

    # Invoke the graph
    final_state = app.invoke(initial_state, config)

    print("--- FINAL AGENT OUTPUT ---")

    for message in final_state["messages"]:
        print(message.content)

    print(
        f"Final Action Taken: {final_state['next_action']}"
    )

Conclusion

Implementing HPA for GPU and CPU workloads requires moving beyond basic CPU thresholds into the realm of custom metrics like VRAM utilization and inference queue depth. However, managing these metrics statically is insufficient for modern, spiky AI workloads.

By wrapping the Kubernetes control plane in an Enterprise Multi-Agent LangGraph RAG system, we achieve:

Context-Aware Scaling: The RAG agent ensures scaling decisions align with historical FinOps budgets and past incident learnings.
Oscillation Prevention: The persistent Checkpointer memory allows the system to remember recent actions, enforcing cooldowns intelligently rather than relying on blind timers.
Stateful Observability: The AgentState provides a complete, traceable audit trail of why a scaling decision was made, bridging the gap between black-box AI and enterprise compliance.

Summary

Modern AI workloads require more than traditional HPA based on CPU or GPU utilization alone. By incorporating custom metrics such as VRAM usage, queue depth, and latency, organizations gain more accurate scaling signals. Combining these metrics with a LangGraph-powered multi-agent RAG architecture enables context-aware, stateful, and memory-driven scaling decisions that improve performance, reduce cloud costs, prevent oscillation, and provide enterprise-grade observability and governance.