In capital markets order execution, the cost of a bad recommendation is measured in basis points of slippage, market impact, and regulatory breach. A system suggesting an aggressive VWAP algorithm during low-liquidity hours may achieve high "acceptance rates" (the vanity metric) while silently destroying execution quality. Conversely, being overly conservative leaves alpha on the table and fails best-execution obligations under MiFID II / SEC Rule 605.

This duality demands two fundamentally different evaluation strategies running in parallel:

This article demonstrates how to architect both within a single LangGraph Multi-Agent RAG System for an institutional equities desk executing across US and EU venues.

The Real-Time Use Case: Smart Order Routing & Algorithm Selection Advisor

Scenario: Traders need real-time guidance when executing large-cap and mid-cap equity orders. Queries include: "Best algo for 250K shares NVDA given current spread and upcoming earnings?" or "Should I use implementation shortfall or TWAP for this FTSE 100 block trade with 15% ADV participation target?"

Why Single-Track Evaluation Fails:

The Solution: A multi-agent system where an Online Pre-Trade Guardrail Agent enforces hard constraints per-order, while an Offline TCA Pipeline continuously recalibrates those constraints based on realized execution quality.

Part 1: The Dual-Track Evaluation Framework

DimensionOnline Evaluation (Pre-Trade)Offline Evaluation (Post-Trade)
Latency Budget<50ms (hard requirement)Hours / End-of-Day batch
Primary GoalPrevent catastrophic execution, ensure complianceOptimize alpha capture, validate strategy
MetricsLiquidity adequacy, max participation rate, venue eligibility, regulatory flagsImplementation shortfall (bps), arrival price vs. VWAP, reversion analysis
Action on FailureBlock recommendation, suggest safer alternativeFlag for quant review, adjust online parameters
Data ScopeLive L2 book, real-time volatility, order attributesHistorical fills, benchmark curves, market microstructure features
Feedback LoopImmediate (within order lifecycle)Periodic (daily/weekly parameter updates)

Critical Insight: Online evaluation is a risk constraint. Offline evaluation is an alpha optimizer. Confusing them leads to either unsafe real-time execution or stagnant strategies that lose edge.

Part 2: Data Representation for Execution Context

Market State & Order Profile (ChromaDB Collection: execution_context)

context_document = {
    "id": "order_nvd_20260812_001",
    "page_content": "NVDA 250,000 shares BUY. Current spread: $0.03 (0.02%). 
                     30-day ADV: 42M shares. Realized vol (20d): 38%. 
                     Earnings release: post-market today. L1 depth top-of-book: 1,200 shares. 
                     Dark pool midpoint liquidity: 18,000 shares visible.",
    "metadata": {
        "type": "order_context",
        "symbol": "NVDA",
        "side": "BUY",
        "quantity": 250000,
        "adv_30d": 42000000,
        "participation_pct_of_adv": 0.60,      # 250K / 42M
        "spread_bps": 2.0,
        "realized_vol_20d_pct": 38.0,
        "top_of_book_depth": 1200,
        "dark_pool_midpoint_liq": 18000,
        "earnings_event": True,
        "event_timing": "post_market_today",
        "venue_eligibility": ["NYSE", "ARCA", "SIGMA_X", "IEX"],
        "regulatory_flags": ["short_sale_restricted"]
    }
}

Execution Strategy & Algo Corpus (ChromaDB Collection: execution_strategies)

strategy_document = {
    "id": "algo_impl_shortfall_aggressive",
    "page_content": "Implementation Shortfall Aggressive: Minimizes delay cost by 
                     front-loading execution. Suitable for high-volatility names with 
                     information advantage. Max participation: 15% ADV. 
                     Requires minimum TOB depth of 5x order slice size. 
                     Not recommended within 2 hours of earnings events.",
    "metadata": {
        "type": "execution_algo",
        "algo_class": "implementation_shortfall",
        "aggression": "high",
        "max_participation_pct_adv": 15.0,
        "min_tob_depth_multiplier": 5,
        "earnings_blackout_hours": 2,
        "suitable_vol_regime": ["high", "extreme"],
        "historical_is_bps_avg": 4.2,         # Backtested average
        "best_execution_compliant": True
    }
}
419

Offline TCA Store (PostgreSQL / Analytics Warehouse)

-- Not in ChromaDB. Used exclusively for offline evaluation.CREATE TABLE execution_recommendation_outcomes (
    recommendation_id UUID PRIMARY KEY,
    order_id VARCHAR(32),
    symbol VARCHAR(10),
    recommended_algo VARCHAR(50),
    online_risk_score FLOAT,             -- Captured at recommendation time
    recommended_participation_pct FLOAT,
    actual_fill_price DECIMAL(12,6),
    arrival_price DECIMAL(12,6),
    vwap_benchmark DECIMAL(12,6),
    implementation_shortfall_bps DECIMAL(8,4),
    reversion_bps DECIMAL(8,4),          -- Post-fill price reversion (alpha signal)
    fill_duration_sec INT,
    venue_breakdown JSONB,
    created_at TIMESTAMP,
    tca_completed_at TIMESTAMP
);

Part 3: Multi-Agent Architecture with Dual-Track Integration

Step 1: Unified State with Online Evaluation Fields

from typing import Annotated, List, Dict, Optional, Literalfrom typing_extensions import TypedDict
from langgraph.graph.message import add_messages
from langchain_core.documents import Document

class ExecutionRecState(TypedDict):
    messages: Annotated[List, add_messages]
    order_id: str
    
    # Context
    order_context: Optional[Document]
    candidate_algos: List[Document]
    
    # === ONLINE EVALUATION STATE ===
    liquidity_adequate: bool
    participation_within_limits: bool
    event_blackout_clear: bool
    regulatory_compliant: bool
    online_risk_score: float                 # 0-1, computed inline
    guardrail_decision: Literal["pass", "fail", "conditional"]
    guardrail_reasoning: str
    
    # Output
    recommendation: Dict
    agent_trace: List[str]

Step 2: Online Pre-Trade Guardrail Agent (<50ms Hard Constraint)

This agent enforces deterministic risk limits derived from offline-optimized parameters loaded at graph initialization. LLMs are NOT used here—pure code path for latency compliance.

import time
from dataclasses import dataclass

@dataclassclass ExecutionThresholds:
    """Loaded from offline TCA pipeline at startup. Never hardcoded."""
    max_participation_pct_adv: float = 15.0
    min_tob_depth_multiplier: int = 3
    earnings_blackout_hours: int = 2
    min_online_risk_score: float = 0.60
    max_spread_bps_for_aggressive: float = 10.0

# Loaded once at service start; refreshed daily via offline pipeline
THRESHOLDS: ExecutionThresholds = load_execution_thresholds()

def online_guardrail_node(state: ExecutionRecState):
    """Deterministic pre-trade risk check. NO LLM. Sub-50ms SLA."""
    start_ns = time.perf_counter_ns()
    
    ctx = state["order_context"].metadata if state["order_context"] else {}
    candidates = state["candidate_algos"]
    
    issues = []
    
    # 1. Liquidity adequacy check
    tob_depth = ctx.get("top_of_book_depth", 0)
    order_qty = ctx.get("quantity", 0)
    min_required = order_qty * THRESHOLDS.min_tob_depth_multiplier / 100  # Slice-based
    liquidity_ok = tob_depth >= min_required
    if not liquidity_ok:
        issues.append(f"TOB depth {tob_depth} < required {min_required:.0f}")
    
    # 2. Participation rate check
    participation = ctx.get("participation_pct_of_adv", 0)
    participation_ok = participation <= THRESHOLDS.max_participation_pct_adv
    if not participation_ok:
        issues.append(f"Participation {participation:.1f}% > limit {THRESHOLDS.max_participation_pct_adv}%")
    
    # 3. Event blackout check
    earnings_event = ctx.get("earnings_event", False)
    blackout_clear = True
    if earnings_event:
        # In production: parse event_timing against current timestamp
        blackout_clear = False  # Simplified; real impl parses timing
        issues.append("Within earnings blackout window")
    
    # 4. Regulatory compliance
    reg_flags = ctx.get("regulatory_flags", [])
    reg_compliant = "short_sale_restricted" not in reg_flags or \
                    all("SELL" != ctx.get("side") for _ in [1])
    if not reg_compliant:
        issues.append("Short sale restriction active on SELL side")
    
    # Composite risk score (deterministic formula, not LLM)
    scores = [
        1.0 if liquidity_ok else 0.2,
        1.0 if participation_ok else max(0, 1.0 - (participation / THRESHOLDS.max_participation_pct_adv - 1)),
        1.0 if blackout_clear else 0.0,
        1.0 if reg_compliant else 0.0
    ]
    risk_score = sum(scores) / len(scores)
    
    decision = "pass" if risk_score >= THRESHOLDS.min_online_risk_score and not issues else "fail"
    if issues and risk_score >= THRESHOLDS.min_online_risk_score:
        decision = "conditional"
    
    elapsed_us = (time.perf_counter_ns() - start_ns) / 1000
    
    return {
        "liquidity_adequate": liquidity_ok,
        "participation_within_limits": participation_ok,
        "event_blackout_clear": blackout_clear,
        "regulatory_compliant": reg_compliant,
        "online_risk_score": round(risk_score, 3),
        "guardrail_decision": decision,
        "guardrail_reasoning": "; ".join(issues) if issues else "All checks passed",
        "agent_trace": state.get("agent_trace", []) + [
            f"guardrail: decision={decision}, risk={risk_score:.3f}, latency={elapsed_us:.0f}us"
        ]
    }

Step 3: Conditional Routing Based on Online Evaluation

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(ExecutionRecState)

workflow.add_node("load_order_context", load_order_context_node)
workflow.add_node("retrieve_algos", algo_retriever_node)
workflow.add_node("online_guardrail", online_guardrail_node)
workflow.add_node("synthesize_rec", execution_synthesizer_node)       # LLM-based explanation
workflow.add_node("safe_fallback", conservative_algo_fallback_node)   # Deterministic safe default
workflow.add_node("log_pre_trade", log_pre_trade_event_node)          # ← Feeds offline TCA

workflow.add_edge(START, "load_order_context")
workflow.add_edge("load_order_context", "retrieve_algos")
workflow.add_edge("retrieve_algos", "online_guardrail")

def guardrail_router(state: ExecutionRecState):
    if state["guardrail_decision"] == "pass":
        return "synthesize_rec"
    elif state["guardrail_decision"] == "conditional":
        return "synthesize_rec"  # Synthesizer adds explicit warnings
    else:
        return "safe_fallback"

workflow.add_conditional_edges("online_guardrail", guardrail_router)
workflow.add_edge("synthesize_rec", "log_pre_trade")
workflow.add_edge("safe_fallback", "log_pre_trade")
workflow.add_edge("log_pre_trade", END)

from langgraph.checkpoint.postgres import PostgresSaver
app = workflow.compile(checkpointer=PostgresSaver.from_conn_string("postgresql://..."))

Step 4: Offline TCA & Parameter Calibration Pipeline

This runs as a scheduled job (EOD + weekly deep analysis). It consumes logged pre-trade events AND post-trade fills to recalibrate online thresholds.

import pandas as pd
import numpy as np
from sqlalchemy import create_engine
from datetime import datetime

engine = create_engine("postgresql://...")

def run_offline_tca_calibration():
    """Daily EOD job: correlate online risk scores with realized execution quality."""
    
    # 1. Join pre-trade recommendations with post-trade TCA
    query = """
        SELECT p.recommended_algo, p.online_risk_score, p.recommended_participation_pct,
               o.implementation_shortfall_bps, o.reversion_bps, o.fill_duration_sec,
               o.symbol, p.created_at
        FROM pre_trade_recommendations p
        JOIN execution_recommendation_outcomes o ON p.recommendation_id = o.recommendation_id
        WHERE o.tca_completed_at >= NOW() - INTERVAL '7 days'
          AND o.implementation_shortfall_bps IS NOT NULL
    """
    df = pd.read_sql(query, engine)
    
    if len(df) < 100:
        log_warning("Insufficient sample size for calibration; skipping update")
        return
    
    # 2. Validate online risk score predicts execution quality
    # Higher risk score should correlate with LOWER implementation shortfall
    corr = df["online_risk_score"].corr(df["implementation_shortfall_bps"])
    
    # 3. Detect miscalibration: high-risk-score orders with HIGH slippage
    miscalibrated = df[
        (df["online_risk_score"] > 0.8) & 
        (df["implementation_shortfall_bps"] > 8.0)
    ]
    
    # 4. Reversion analysis: are we capturing alpha or paying for urgency?
    avg_reversion_by_algo = df.groupby("recommended_algo")["reversion_bps"].mean()
    
    # 5. Generate updated thresholds
    current = load_execution_thresholds()
    updated = current.copy()
    
    if len(miscalibrated) > len(df) * 0.05:  # >5% miscalibration rate
        # Tighten risk threshold
        updated.min_online_risk_score = min(0.90, current.min_online_risk_score + 0.05)
        # Reduce max participation
        updated.max_participation_pct_adv = max(5.0, current.max_participation_pct_adv - 2.0)
        
        send_quant_alert(
            f"Miscalibration detected: {len(miscalibrated)} high-risk/high-slippage orders. "
            f"Tightening thresholds: risk_min={updated.min_online_risk_score}, "
            f"max_part={updated.max_participation_pct_adv}%"
        )
    
    # 6. Alpha attribution: flag algos with negative reversion consistently
    negative_alpha_algos = avg_reversion_by_algo[avg_reversion_by_algo < -2.0].index.tolist()
    if negative_alpha_algos:
        send_quant_alert(
            f"Negative alpha detected for algos: {negative_alpha_algos}. "
            f"Reversion bps: {avg_reversion_by_algo[negative_alpha_algos].to_dict()}"
        )
        # DO NOT auto-disable; flag for quant desk review
    
    # 7. Persist updated thresholds atomically
    save_execution_thresholds(updated)
    
    log_calibration_run(
        timestamp=datetime.utcnow(),
        samples=len(df),
        risk_slippage_correlation=corr,
        miscalibration_rate=len(miscalibrated)/len(df),
        old_thresholds=current.__dict__,
        new_thresholds=updated.__dict__,
        negative_alpha_algos=negative_alpha_algos
    )
    
    return updated

Part 5: Closing the Loop Safely

The critical architectural constraint: offline outputs NEVER directly modify online execution parameters without validation.

def deploy_updated_thresholds(new_thresholds: ExecutionThresholds):
    """Human-in-the-loop + shadow validation before live deployment."""
    
    # 1. Shadow test: replay last 24h of orders against new thresholds
    shadow_results = simulate_guardrail(new_thresholds, historical_orders_24h)
    
    # 2. Check: would new thresholds have blocked profitable trades?
    missed_alpha = shadow_results["blocked_orders_with_positive_reversion"]
    if missed_alpha > 20:
        raise DeploymentBlockError(
            f"Threshold change would have blocked {missed_alpha} alpha-positive orders. "
            "Requires quant desk sign-off."
        )
    
    # 3. Check: does new threshold maintain best-execution compliance?
    if new_thresholds.max_participation_pct_adv < 3.0:
        raise DeploymentBlockError(
            "Participation limit too restrictive; may violate best-execution obligation "
            "for illiquid names."
        )
    
    # 4. Atomic deploy with rollback capability
    atomic_threshold_update(new_thresholds)
    notify_trading_desk(f"Execution parameters updated: {new_thresholds.__dict__}")

Production Considerations for Capital Markets

  1. Latency Is Non-Negotiable: The online guardrail MUST be pure deterministic code. No LLM, no vector search, no external API calls in the hot path. All context must be pre-loaded into memory at service start. Target p99 < 50μs.

  2. Regulatory Immutability: Some constraints (e.g., short-sale restrictions, venue eligibility under Reg NMS/MiFID II) are hard-coded constants, never adjusted by offline TCA. Separate configurable parameters from immutable regulatory rules in your threshold schema.

  3. TCA Benchmark Integrity: Offline evaluation is only as good as your benchmarks. Validate arrival price, VWAP, and implementation shortfall calculations against independent third-party TCA providers quarterly. Discrepancies invalidate all calibration.

  4. Market Regime Detection: Offline calibration must be stratified by volatility regime and liquidity tier. Parameters optimized for low-vol large-cap fail catastrophically during earnings season or in small-cap names. Maintain regime-specific threshold sets.

  5. Audit Trail for Best Execution: Every online guardrail decision, every offline calibration run, and every threshold deployment must be immutably logged with timestamps and rationale. Regulators will demand proof that your execution recommendations were systematically validated and continuously improved.

  6. Trader Override Tracking: When traders override system recommendations, log BOTH the system recommendation and the override outcome. Overrides are the most valuable signal for recalibrating trust boundaries and detecting model blind spots.

Conclusion

In capital markets order execution, online and offline evaluation aren't alternatives—they're complementary halves of a best-execution system. Online guardrails prevent immediate execution damage using the best available real-time risk parameters. Offline TCA ensures those parameters evolve with market microstructure reality, detecting miscalibration and alpha decay before they compound into systematic underperformance. The LangGraph architecture makes this duality native: the online graph enforces sub-millisecond risk constraints per-order, while structured event logging feeds the offline TCA pipeline that continuously refines those constraints. Neither operates in isolation. Together, they form a system that is simultaneously safe in the moment and sharper over time—which is precisely what best-execution obligations demand.