Measuring Recommendation Quality in Treasury Cash Forecasting with LangGraph RAG

In corporate treasury management, Click-Through Rate (CTR) is not just irrelevant—it’s financially meaningless. A treasurer clicking on a "Recommended Cash Pool Transfer" doesn’t indicate value; only whether that transfer actually prevented a shortfall or optimized yield without breaching covenants matters. Treasury recommendations are high-stakes decisions where false positives cause liquidity crises and false negatives leave millions earning zero basis points. This article demonstrates how to build and evaluate a Cash Forecast Recommendation Engine where quality measurement is embedded directly into the LangGraph orchestration layer, using ChromaDB for contextual retrieval, persistent state for auditability, and a dual-track evaluation framework aligned with treasury KPIs.

The Real-Time Use Case: Daily Liquidity Position Advisor

Scenario: A multinational corporation with 47 subsidiaries across 12 currencies needs daily cash position recommendations. Treasurers ask: "What’s our optimal EUR pooling strategy given next week’s APAC payroll and pending FX hedges?" or "Should we draw down the revolver or liquidate MMF holdings to cover the projected Day+7 deficit?"

Why CTR Fails Here:

Clicks don’t correlate with forecast accuracy or funding cost savings
Recommendations may be clicked but overridden due to unmodeled covenant constraints
Silent failures (ignored recommendations that would have prevented shortfalls) are invisible to CTR

The Solution: A multi-agent system where a Treasury Quality Evaluator computes a composite Forecast Recommendation Quality Score (FRQS) across four financially-grounded dimensions before any recommendation surfaces to the treasurer.

Part 1: The Four-Dimension FRQS Framework

Dimension	Metric	Treasury Impact
Forecast Accuracy Proxy	Historical MAPE of similar scenario recommendations	Directly correlates with liquidity risk
Covenant & Policy Compliance	Binary pass/fail + margin buffer to breach thresholds	Prevents technical defaults and penalties
Economic Value	Net interest saved / opportunity cost captured vs. baseline	Quantifiable P&L impact
Execution Feasibility	Settlement window alignment + counterparty availability + system readiness	Ensures recommendations are actionable, not theoretical

These are computed per-recommendation in real-time and persisted for backtesting against actual cash flows.

Part 2: Data Representation for Treasury Context

Cash Position & Covenant State (ChromaDB Collection: `treasury_context`)

context_document = {
    "id": "cash_pos_eur_20260812",
    "page_content": "EUR consolidated position: €42.3M cash, €18.7M MMF maturing Aug 15. 
                     Projected Day+7 net outflow: €31.2M (APAC payroll €22M, vendor payments €9.2M). 
                     Revolver headroom: €35M at SOFR+1.2%. Debt covenant: min liquidity ratio 1.25x, 
                     current 1.38x. Last forecast error (similar pattern): +4.2% overestimate.",
    "metadata": {
        "type": "cash_position",
        "currency": "EUR",
        "as_of_date": "2026-08-12",
        "cash_balance_eur": 42300000,
        "mmf_maturing_eur": 18700000,
        "mmf_maturity_date": "2026-08-15",
        "projected_day7_net_flow_eur": -31200000,
        "revolver_headroom_eur": 35000000,
        "revolver_rate_bps_over_sofr": 120,
        "covenant_liquidity_ratio_min": 1.25,
        "covenant_liquidity_ratio_current": 1.38,
        "historical_forecast_mape_pct": 4.2,
        "apac_payroll_date": "2026-08-18",
        "fx_hedge_settlement_date": "2026-08-14"
    }
}

Funding & Investment Options (ChromaDB Collection: `treasury_instruments`)

instrument_document = {
    "id": "inst_eur_mmf_fidelity_govt",
    "page_content": "Fidelity EUR Government MMF: Current yield 3.42% p.a. T+1 settlement. 
                     Min redemption €1M. No early redemption penalty. Eligible for LCR HQLA Level 1. 
                     Counterparty limit remaining: €25M. Last NAV deviation: 0.00%.",
    "metadata": {
        "type": "investment_instrument",
        "instrument_class": "mmf",
        "currency": "EUR",
        "yield_annual_pct": 3.42,
        "settlement_days": 1,
        "min_redemption_eur": 1000000,
        "counterparty_limit_remaining_eur": 25000000,
        "hqla_level": 1,
        "eligible_for_covenant_lcr": True,
        "last_nav_deviation_pct": 0.00
    }
}

Part 3: Multi-Agent Architecture with Inline FRQS Evaluation

Step 1: State Schema with FRQS Fields

from typing import Annotated, List, Dict, Optional, Literalfrom typing_extensions import TypedDict
from langgraph.graph.message import add_messages
from langchain_core.documents import Document

class TreasuryRecState(TypedDict):
    messages: Annotated[List, add_messages]
    entity_id: str  # Subsidiary or consolidated entity
    
    # Context
    cash_position: Optional[Document]
    candidate_instruments: List[Document]
    
    # === ONLINE VALIDATION STATE ===
    covenant_compliant: bool
    covenant_margin_pct: float             # Buffer above breach threshold
    execution_feasible: bool
    settlement_window_ok: bool
    
    # === FRQS EVALUATION STATE ===
    forecast_accuracy_score: float         # 0-1 based on historical MAPE
    compliance_score: float                # 0-1 based on covenant margin
    economic_value_score: float            # 0-1 normalized interest saved
    feasibility_score: float               # 0-1 based on execution constraints
    overall_frqs: float                    # Weighted composite
    frqs_reasoning: str
    
    # Output
    recommendation: Dict
    agent_trace: List[str]

Step 2: Covenant Compliance Validator (Hard Gate)

This node performs deterministic financial math before any LLM involvement. Covenant breaches are non-negotiable.

def covenant_validator_node(state: TreasuryRecState):
    """Deterministic covenant check. NO LLM. Zero tolerance for hallucination."""
    pos = state["cash_position"]
    if not pos:
        return {"covenant_compliant": False, "covenant_margin_pct": 0.0}
    
    meta = pos.metadata
    
    # Example: Liquidity Ratio Covenant = (Cash + HQLA) / Short-Term Obligations
    # In production, fetch real-time obligations from ERP
    current_ratio = meta["covenant_liquidity_ratio_current"]
    min_ratio = meta["covenant_liquidity_ratio_min"]
    margin_pct = (current_ratio - min_ratio) / min_ratio * 100
    
    compliant = current_ratio >= min_ratio
    
    return {
        "covenant_compliant": compliant,
        "covenant_margin_pct": round(margin_pct, 2),
        "agent_trace": state.get("agent_trace", []) + [
            f"covenant_check: ratio={current_ratio:.3f}, min={min_ratio}, "
            f"margin={margin_pct:.1f}%, compliant={compliant}"
        ]
    }

Step 3: Execution Feasibility Agent

Checks real-world constraints that make recommendations actionable.

from datetime import datetime, timedelta

async def feasibility_checker_node(state: TreasuryRecState):
    """Validate settlement windows, counterparty limits, and system cut-offs."""
    feasible = True
    issues = []
    
    today = datetime.utcnow().date()
    pos_meta = state["cash_position"].metadata
    
    for inst in state["candidate_instruments"]:
        meta = inst.metadata
        
        # Settlement window check
        settlement_date = today + timedelta(days=meta["settlement_days"])
        if settlement_date > pos_meta["apac_payroll_date"]:
            feasible = False
            issues.append(f"{meta['instrument_class']} settles after payroll date")
        
        # Counterparty limit check
        if meta.get("counterparty_limit_remaining_eur", 0) < abs(pos_meta["projected_day7_net_flow_eur"]):
            feasible = False
            issues.append(f"Counterparty limit insufficient for {meta['instrument_class']}")
    
    return {
        "execution_feasible": feasible,
        "settlement_window_ok": feasible,
        "agent_trace": state["agent_trace"] + [
            f"feasibility: ok={feasible}, issues={issues}"
        ]
    }

Step 4: FRQS Evaluation Agent (Composite Financial Scoring)

Runs ONLY after hard gates pass. Computes the four-dimension score with treasury-aligned weights.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

frqs_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a Treasury Recommendation Quality Auditor.
    Score this recommendation on FOUR dimensions (0-1 each):
    
    1. FORECAST ACCURACY: Given historical MAPE ({mape}%) for similar scenarios, 
       how reliable is this projection? Lower MAPE = higher score.
    2. COMPLIANCE: Given covenant margin ({margin}%), how safe is this action? 
       Margin >20% = 1.0, 10-20% = 0.7, 5-10% = 0.4, <5% = 0.1.
    3. ECONOMIC VALUE: Net interest saved vs. baseline (revolver draw at {revolver_rate} bps 
       over SOFR). Normalize: €0 saved = 0, ≥€50K saved = 1.0.
    4. FEASIBILITY: Settlement alignment + counterparty limits + system cut-offs. 
       All clear = 1.0, minor issues = 0.6, blocking issues = 0.0.
    
    WEIGHTS: accuracy=0.30, compliance=0.35, value=0.20, feasibility=0.15
    
    Return JSON: {{
        "forecast_accuracy": float, "compliance": float, 
        "economic_value": float, "feasibility": float,
        "overall_frqs": float, "reasoning": string
    }}"""),
    ("human", "Recommendation: {rec}\nContext: {context}")
])

async def frqs_evaluation_node(state: TreasuryRecState):
    if not state["covenant_compliant"] or not state["execution_feasible"]:
        return {
            "overall_frqs": 0.0,
            "frqs_reasoning": "Failed hard gate; skipping FRQS evaluation",
            "recommendation": None
        }
    
    chain = frqs_prompt | llm.with_structured_output(dict)
    result = chain.invoke({
        "mape": state["cash_position"].metadata["historical_forecast_mape_pct"],
        "margin": state["covenant_margin_pct"],
        "revolver_rate": state["cash_position"].metadata["revolver_rate_bps_over_sofr"],
        "rec": "\n---\n".join(d.page_content for d in state["candidate_instruments"]),
        "context": state["cash_position"].page_content
    })
    
    return {
        "forecast_accuracy_score": result["forecast_accuracy"],
        "compliance_score": result["compliance"],
        "economic_value_score": result["economic_value"],
        "feasibility_score": result["feasibility"],
        "overall_frqs": result["overall_frqs"],
        "frqs_reasoning": result["reasoning"],
        "agent_trace": state["agent_trace"] + [f"frqs_eval: score={result['overall_frqs']:.3f}"]
    }

Step 5: Conditional Routing + Outcome Logging

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(TreasuryRecState)

workflow.add_node("load_context", load_treasury_context_node)
workflow.add_node("retrieve_instruments", instrument_retriever_node)
workflow.add_node("validate_covenant", covenant_validator_node)
workflow.add_node("check_feasibility", feasibility_checker_node)
workflow.add_node("evaluate_frqs", frqs_evaluation_node)
workflow.add_node("synthesize", recommendation_synthesizer_node)
workflow.add_node("safe_fallback", conservative_baseline_node)
workflow.add_node("log_outcome_hook", log_frqs_event_node)

workflow.add_edge(START, "load_context")
workflow.add_edge("load_context", "retrieve_instruments")
workflow.add_edge("retrieve_instruments", "validate_covenant")

def covenant_router(state: TreasuryRecState):
    return "check_feasibility" if state["covenant_compliant"] else "safe_fallback"

workflow.add_conditional_edges("validate_covenant", covenant_router)
workflow.add_edge("check_feasibility", "evaluate_frqs")

def frqs_gate(state: TreasuryRecState):
    if state["overall_frqs"] >= 0.75:
        return "synthesize"
    if state.get("retry_count", 0) < 2:
        return "retrieve_instruments"  # Try alternative instruments
    return "safe_fallback"

workflow.add_conditional_edges("evaluate_frqs", frqs_gate)
workflow.add_edge("synthesize", "log_outcome_hook")
workflow.add_edge("safe_fallback", "log_outcome_hook")
workflow.add_edge("log_outcome_hook", END)

from langgraph.checkpoint.postgres import PostgresSaver
app = workflow.compile(checkpointer=PostgresSaver.from_conn_string("postgresql://..."))

Part 4: Offline Backtesting Pipeline

import pandas as pd
from sqlalchemy import create_engine
from sklearn.metrics import mean_absolute_percentage_error

engine = create_engine("postgresql://...")

def backtest_frqs_weekly():
    """Correlate FRQS scores with actual cash flow outcomes."""
    
    df = pd.read_sql("""
        SELECT overall_frqs, forecast_accuracy_score, compliance_score,
               economic_value_score, feasibility_score,
               actual_vs_forecast_error_pct, funding_cost_saved_eur,
               covenant_breached_flag, execution_failed_flag
        FROM treasury_recommendation_outcomes
        WHERE outcome_date >= NOW() - INTERVAL '7 days'
          AND actual_vs_forecast_error_pct IS NOT NULL
    """, engine)
    
    # 1. Validate FRQS predicts forecast accuracy
    mape_by_frqs_bucket = df.groupby(pd.qcut(df["overall_frqs"], q=5))["actual_vs_forecast_error_pct"].mean()
    
    # 2. Economic value validation: did high-value-score recs actually save money?
    value_correlation = df["economic_value_score"].corr(df["funding_cost_saved_eur"])
    
    # 3. Compliance safety check: any breaches in high-compliance-score bucket?
    high_compliance_breaches = df[
        (df["compliance_score"] > 0.9) & (df["covenant_breached_flag"] == True)
    ]
    if len(high_compliance_breaches) > 0:
        alert_risk_committee(
            f"CRITICAL: {len(high_compliance_breaches)} covenant breaches in high-compliance-score recommendations"
        )
    
    # 4. Recalibrate weights if dimension predictive power shifted
    # (Similar to QR payment example, but with treasury-specific validation)
    
    log_backtest_run(mape_by_frqs_bucket, value_correlation, len(high_compliance_breaches))

Production Considerations for Treasury

Deterministic Gates First: Covenant checks and feasibility validation MUST be pure code, never LLM. Use LLMs only for scoring and synthesis after hard constraints are satisfied.
Audit Immutability: Every FRQS computation, covenant check, and recommendation must be cryptographically hashed and stored immutably. Regulators and auditors will demand full decision traceability.
Data Freshness SLAs: Cash positions stale by >15 minutes invalidate all recommendations. Implement real-time bank feed ingestion with freshness checks in the context loader. Reject queries when data is stale.
Human-in-the-Loop Override Tracking: When treasurers override recommendations, log BOTH the system FRQS and the override rationale. Overrides are critical training signal for recalibrating economic value scoring.
Stress Testing Integration: FRQS thresholds should be dynamically adjusted during stress periods (e.g., market volatility spikes). Integrate with your enterprise risk management system to auto-tighten compliance margins during stress.

Conclusion

In treasury cash forecasting, recommendation quality isn’t about engagement—it’s about liquidity preservation, covenant safety, and quantifiable economic value. By embedding a four-dimension FRQS evaluator within the LangGraph orchestration layer, enforcing deterministic hard gates before LLM scoring, and closing the loop with offline backtesting against actual cash flows and funding costs, you build a system that optimizes for what treasury actually cares about: keeping the company solvent and capital working efficiently. CTR tells you what was clicked. FRQS tells you whether that recommendation kept the lights on and saved real money. In treasury, that distinction is the difference between a dashboard toy and a mission-critical financial control.