Introduction

In enterprise machine learning architectures, feature engineering is rarely a monolithic process. It is bifurcated into two distinct modes: Offline Computation for model training and historical analysis, and Online Serving for real-time inference. While offline features provide depth and historical context, online features offer immediacy and relevance. The critical challenge lies in ensuring that these two modes remain consistent—a problem known as "training-serving skew." If the features used to train a model differ significantly from those served during prediction, model performance degrades rapidly.

This article explores how to harmonize online and offline feature pipelines using an enterprise-grade Multi-Agent Retrieval-Augmented Generation (RAG) system built on LangGraph. By treating feature consistency as a stateful, multi-agent problem, we can create systems that not only serve low-latency predictions but also continuously validate their data integrity against historical baselines. This approach ensures that real-time decisions are both fast and trustworthy.

The Dual Nature of Feature Engineering: Online vs. Offline

Key Challenges: Training-Serving Skew and Latency Constraints

Real-Time Use Case: Dynamic Fraud Detection in Fintech

Scenario: "PaySecure," a digital payment processor, needs to detect fraudulent transactions in real-time. The model relies on:

  1. Offline Features: User’s historical average spend, typical merchant categories, and long-term risk score (computed daily).

  2. Online Features: Current transaction amount, location velocity (distance from last transaction), and device fingerprint (computed in milliseconds).

Problem: During a high-volume sales event, the online serving layer experiences latency spikes, causing some features to time out. The system must decide whether to use stale cached values, default values, or block the transaction entirely, all while maintaining consistency with the offline-trained model’s expectations.

Objective: Build a Multi-Agent LangGraph system that:

  1. Retrieves online features from a low-latency store.

  2. Fetches offline baseline features for context.

  3. Uses RAG to retrieve fraud policy rules and historical skew reports.

  4. Makes a reasoned decision on how to handle missing or stale online features.

  5. Maintains state to track the decision lineage for audit purposes.

Architecture Overview: Hybrid Feature Pipeline with Multi-Agent Orchestration

We use LangGraph to orchestrate the interaction between online and offline data sources. The workflow includes:

Step-by-Step POC Implementation

Backend: Offline Batch Processor and Online Serving Layer

We simulate offline and online stores using Python dictionaries. In production, these would be PostgreSQL and Redis respectively.

# backend/feature_store.py
from pydantic import BaseModel
from typing import Dict, Optional
import time

class OnlineFeatures(BaseModel):
    transaction_amount: float
    location_velocity_km_h: float
    device_trust_score: float

class OfflineFeatures(BaseModel):
    avg_spend_90d: float
    typical_merchant_category: str
    long_term_risk_score: float

class MockOnlineStore:
    def __init__(self):
        self.store = {}
        
    def set_features(self, user_id: str, features: OnlineFeatures):
        self.store[user_id] = {"data": features.dict(), "ts": time.time()}
        
    def get_features(self, user_id: str) -> Optional[Dict]:
        return self.store.get(user_id)

class MockOfflineStore:
    def __init__(self):
        self.store = {
            "USER_001": OfflineFeatures(
                avg_spend_90d=150.0, 
                typical_merchant_category="electronics", 
                long_term_risk_score=0.2
            )
        }
        
    def get_features(self, user_id: str) -> Optional[OfflineFeatures]:
        return self.store.get(user_id)

online_store = MockOnlineStore()
offline_store = MockOfflineStore()

# Seed online data
online_store.set_features("USER_001", OnlineFeatures(
    transaction_amount=1200.0, 
    location_velocity_km_h=800.0, 
    device_trust_score=0.9
))

Multi-Agent Graph: Stateful Reconciliation and RAG Context

The graph orchestrates the retrieval and decision-making process.

# backend/graph.py
from typing import TypedDict, Annotated, List, Optional
from langgraph.graph import StateGraph, END
from langchain_core.messages import HumanMessage, AIMessage
import operator
from .feature_store import online_store, offline_store

class FraudDetectionState(TypedDict):
    user_id: str
    online_features: Optional[dict]
    offline_features: Optional[dict]
    skew_detected: bool
    policy_context: List[str]
    decision: str
    messages: Annotated[List, operator.add]

def fetch_online_features(state: FraudDetectionState):
    """Agent 1: Fetches real-time features"""
    raw = online_store.get_features(state["user_id"])
    if not raw:
        return {"online_features": None, "messages": [AIMessage(content="Online features missing")]}
    # Check for staleness (e.g., older than 5 seconds)
    if time.time() - raw["ts"] > 5:
        return {"online_features": None, "messages": [AIMessage(content="Online features stale")]}
    return {"online_features": raw["data"], "messages": [AIMessage(content="Online features retrieved")]}

def fetch_offline_baseline(state: FraudDetectionState):
    """Agent 2: Fetches historical baseline"""
    offline = offline_store.get_features(state["user_id"])
    if not offline:
        return {"offline_features": None, "messages": [AIMessage(content="Offline baseline missing")]}
    return {"offline_features": offline.dict(), "messages": [AIMessage(content="Offline baseline retrieved")]}

def check_skew_and_retrieve_policy(state: FraudDetectionState):
    """Agent 3: Detects skew and retrieves RAG context"""
    online = state["online_features"]
    offline = state["offline_features"]
    
    skew = False
    if online and offline:
        # Simple skew check: if current amount is > 5x average
        if online["transaction_amount"] > (offline["avg_spend_90d"] * 5):
            skew = True
            
    # Mock RAG retrieval for policies
    policies = ["Policy A: High velocity + High amount = Manual Review", "Policy B: Stale features = Block"]
    
    return {"skew_detected": skew, "policy_context": policies, "messages": [AIMessage(content=f"Skew detected: {skew}")] }

def make_fraud_decision(state: FraudDetectionState):
    """Agent 4: Final decision based on state"""
    if not state["online_features"]:
        decision = "BLOCK: Missing/Stale Online Data"
    elif state["skew_detected"]:
        decision = "REVIEW: High Value Anomaly Detected"
    else:
        decision = "APPROVE: Normal Pattern"
        
    return {"decision": decision, "messages": [AIMessage(content=decision)]}

# Build Graph
workflow = StateGraph(FraudDetectionState)
workflow.add_node("fetch_online", fetch_online_features)
workflow.add_node("fetch_offline", fetch_offline_baseline)
workflow.add_node("check_skew", check_skew_and_retrieve_policy)
workflow.add_node("decide", make_fraud_decision)

workflow.set_entry_point("fetch_online")
workflow.add_edge("fetch_online", "fetch_offline")
workflow.add_edge("fetch_offline", "check_skew")
workflow.add_edge("check_skew", "decide")
workflow.add_edge("decide", END)

app = workflow.compile()

Frontend: Feature Health and Latency Monitor

A React dashboard displays the feature values, skew status, and final decision.

// frontend/src/components/FraudMonitor.jsx
import { useState } from 'react';

export default function FraudMonitor() {
  const [userId, setUserId] = useState('USER_001');
  const [result, setResult] = useState(null);

  const handleCheck = async () => {
    const response = await fetch(`/api/fraud-check?user_id=${userId}`);
    const data = await response.json();
    setResult(data);
  };

  return (
    <div className="p-6 max-w-3xl mx-auto">
      <h2 className="text-2xl font-bold mb-4">Fraud Detection Feature Monitor</h2>
      <input 
        type="text" 
        value={userId} 
        onChange={(e) => setUserId(e.target.value)}
        className="border p-2 mr-2"
      />
      <button onClick={handleCheck} className="bg-red-600 text-white p-2 rounded">
        Run Fraud Check
      </button>
      
      {result && (
        <div className="mt-6 space-y-4">
          <div className="grid grid-cols-2 gap-4">
            <div className="bg-blue-50 p-4 rounded">
              <h3 className="font-semibold">Online Features</h3>
              <pre>{JSON.stringify(result.online_features, null, 2)}</pre>
            </div>
            <div className="bg-gray-50 p-4 rounded">
              <h3 className="font-semibold">Offline Baseline</h3>
              <pre>{JSON.stringify(result.offline_features, null, 2)}</pre>
            </div>
          </div>
          
          <div className={`p-4 rounded border-l-4 ${result.decision.includes('BLOCK') ? 'bg-red-100 border-red-500' : result.decision.includes('REVIEW') ? 'bg-yellow-100 border-yellow-500' : 'bg-green-100 border-green-500'}`}>
            <h3 className="font-semibold">Final Decision</h3>
            <p className="text-lg">{result.decision}</p>
          </div>
          
          <details className="bg-white p-4 rounded shadow">
            <summary className="cursor-pointer font-medium">View Agent Trace</summary>
            <ul className="mt-2 list-disc pl-5">
              {result.messages.map((m, i) => (
                <li key={i} className="text-sm text-gray-700">{m.content}</li>
              ))}
            </ul>
          </details>
        </div>
      )}
    </div>
  );
}

Conclusion

Handling the dichotomy between online serving and offline computation is one of the most significant hurdles in enterprise MLOps. While offline features provide the statistical depth required for robust model training, online features provide the temporal relevance needed for real-time action. By implementing a Multi-Agent LangGraph architecture, organizations can bridge this gap effectively. This system not only serves features but also actively monitors for training-serving skew and applies contextual policies to manage data inconsistencies. As AI systems become more autonomous, the ability to maintain this balance will be crucial for ensuring both performance and reliability in production environments.