In microfinance repayment tracking, the cost of a bad recommendation isn't a lost click—it's default risk, regulatory violation, or borrower distress. A system suggesting "increase installment by 20%" to a farmer during drought season may optimize short-term recovery but destroy long-term portfolio health. Conversely, being overly conservative leaves capital idle and undermines institutional sustainability.

This duality demands two fundamentally different evaluation strategies running in parallel:

This article demonstrates how to architect both within a single LangGraph Multi-Agent RAG System for a Microfinance Institution (MFI) serving 200K+ borrowers across agricultural and SME segments.

The Real-Time Use Case: Dynamic Repayment Rescheduling Advisor

Scenario: Loan officers and borrower-facing apps need real-time guidance when borrowers signal repayment difficulty. Queries include: "Borrower #MF-8847 missed 2 payments, rice harvest delayed 3 weeks" or "SME borrower requests 60-day grace period, cash flow shows seasonal dip."

Why Single-Track Evaluation Fails:

The Solution: A multi-agent system where an Online Guardrail Agent enforces hard constraints per-interaction, while an Offline Evaluation Pipeline continuously recalibrates those constraints based on portfolio outcomes.

Part 1: The Dual-Track Evaluation Framework

DimensionOnline Evaluation (Real-Time)Offline Evaluation (Batch)
Latency Budget<500msHours/Days
Primary GoalPrevent harm, ensure complianceOptimize policy, detect drift
MetricsRegulatory pass/fail, affordability ratio, data completenessDefault rate Δ, recovery rate, fairness disparity index
Action on FailureBlock recommendation, return safe fallbackFlag for human review, adjust online thresholds
Data ScopeCurrent borrower state + active regulationsHistorical cohort outcomes + macro indicators
Feedback LoopImmediate (within session)Periodic (weekly/monthly model updates)

Critical Insight: Online evaluation is a constraint. Offline evaluation is an optimizer. Confusing them leads to either unsafe real-time decisions or stagnant policies.

Part 2: Data Representation for Repayment Context

Borrower Profile (ChromaDB Collection: borrower_profiles)

borrower_document = {
    "id": "borrower_MF_8847",
    "page_content": "Agricultural borrower, rice cultivation, Region IV-A. 
                     Loan cycle 3, previous cycles fully repaid on time. 
                     Current outstanding: PHP 28,500. Missed 2 installments. 
                     Last income verification: 2026-07-15. Household size: 5.",
    "metadata": {
        "type": "borrower",
        "segment": "agricultural",
        "loan_cycle": 3,
        "outstanding_balance_php": 28500,
        "missed_installments": 2,
        "days_past_due": 18,
        "last_income_verification_date": "2026-07-15",
        "region": "IV-A",               # Drought/flood risk zone
        "previous_default_history": 0,
        "household_dependents": 5
    }
}

Repayment Policy & Regulation Corpus (ChromaDB Collection: repayment_policies)

policy_document = {
    "id": "policy_bsp_restructuring_2026",
    "page_content": "BSP Circular 2026-045: Agricultural loan restructuring 
                     permitted when crop failure documented. Max grace period: 90 days. 
                     Mandatory affordability assessment: revised installment ≤ 40% of 
                     verified monthly household income. Requires barangay certification 
                     for force majeure claims.",
    "metadata": {
        "type": "regulation",
        "jurisdiction": "PH",
        "effective_date": "2026-03-01",
        "applicable_segments": ["agricultural"],
        "max_grace_period_days": 90,
        "affordability_threshold_pct": 40,
        "requires_documentation": ["barangay_cert", "income_verification"]
    }
}

Offline Outcome Store (PostgreSQL / Analytics Warehouse)

-- Not in ChromaDB. Used exclusively for offline evaluation.
CREATE TABLE repayment_recommendation_outcomes (
    recommendation_id UUID PRIMARY KEY,
    borrower_id VARCHAR(32),
    recommended_action VARCHAR(50),      -- 'grace_period', 'restructure', 'refinance'
    online_risk_score FLOAT,             -- Captured at recommendation time
    actual_outcome VARCHAR(20),          -- 'repaid', 'partial', 'default', 'still_active'
    days_to_outcome INT,
    recovery_amount_php DECIMAL(12,2),
    created_at TIMESTAMP,
    outcome_recorded_at TIMESTAMP
);

Part 3: Multi-Agent Architecture with Dual-Track Integration

415

Step 1: Unified State with Online Evaluation Fields

from typing import Annotated, List, Dict, Optional, Literal
from typing_extensions import TypedDict
from langgraph.graph.message import add_messages
from langchain_core.documents import Document

class RepaymentRecState(TypedDict):
    messages: Annotated[List, add_messages]
    borrower_id: str
    borrower_profile: Optional[Document]
    
    # Retrieval
    relevant_policies: List[Document]
    
    # === ONLINE EVALUATION STATE ===
    affordability_ratio: Optional[float]     # Revised installment / verified income
    documentation_complete: bool
    regulatory_compliant: bool
    online_risk_score: float                 # 0-1, computed inline
    guardrail_decision: Literal["pass", "fail", "conditional"]
    guardrail_reasoning: str
    
    # Output
    recommendation: Dict
    agent_trace: List[str]

Step 2: Online Guardrail Agent (Real-Time Safety Gate)

This agent enforces hard constraints derived from offline-optimized thresholds loaded at graph initialization.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
import json

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Thresholds loaded from offline evaluation pipeline at startup
# These are UPDATED periodically, not hardcoded
ONLINE_THRESHOLDS = load_thresholds_from_config()  
# e.g., {"max_affordability_ratio": 0.40, "min_risk_score_pass": 0.65, 
#         "required_docs_agricultural": ["barangay_cert"]}

guardrail_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a Microfinance Repayment Compliance Auditor.
    Evaluate this proposed rescheduling against BSP regulations and internal policy.
    
    THRESHOLDS (updated via offline evaluation):
    - Max affordability ratio: {max_affordability}
    - Min risk score to pass: {min_risk_score}
    - Required documentation for {segment}: {required_docs}
    
    BORROWER STATE: {profile}
    APPLICABLE POLICIES: {policies}
    PROPOSED ACTION: {proposed}
    
    Return JSON: {{
        "affordability_ratio": float or null,
        "documentation_complete": bool,
        "regulatory_compliant": bool,
        "online_risk_score": float,
        "decision": "pass" | "fail" | "conditional",
        "reasoning": string
    }}"""),
    ("human", "{query}")
])

def online_guardrail_node(state: RepaymentRecState):
    chain = guardrail_prompt | llm.with_structured_output(dict)
    
    result = chain.invoke({
        "max_affordability": ONLINE_THRESHOLDS["max_affordability_ratio"],
        "min_risk_score": ONLINE_THRESHOLDS["min_risk_score_pass"],
        "segment": state["borrower_profile"].metadata["segment"] if state["borrower_profile"] else "unknown",
        "required_docs": ONLINE_THRESHOLDS.get(
            f"required_docs_{state['borrower_profile'].metadata.get('segment', 'general')}", []
        ),
        "profile": state["borrower_profile"].page_content if state["borrower_profile"] else "No profile",
        "policies": "\n---\n".join(p.page_content for p in state["relevant_policies"]),
        "proposed": state["messages"][-1].content,
        "query": state["messages"][-1].content
    })
    
    return {
        "affordability_ratio": result["affordability_ratio"],
        "documentation_complete": result["documentation_complete"],
        "regulatory_compliant": result["regulatory_compliant"],
        "online_risk_score": result["online_risk_score"],
        "guardrail_decision": result["decision"],
        "guardrail_reasoning": result["reasoning"],
        "agent_trace": state["agent_trace"] + [
            f"guardrail: decision={result['decision']}, risk={result['online_risk_score']:.3f}"
        ]
    }

Step 3: Conditional Routing Based on Online Evaluation

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(RepaymentRecState)

workflow.add_node("load_borrower", load_borrower_node)
workflow.add_node("retrieve_policies", policy_retriever_node)
workflow.add_node("online_guardrail", online_guardrail_node)
workflow.add_node("synthesize", recommendation_synthesizer_node)
workflow.add_node("safe_fallback", regulatory_safe_fallback_node)
workflow.add_node("log_event", log_recommendation_event_node)  # ← Feeds offline pipeline

workflow.add_edge(START, "load_borrower")
workflow.add_edge("load_borrower", "retrieve_policies")
workflow.add_edge("retrieve_policies", "online_guardrail")

def guardrail_router(state: RepaymentRecState):
    if state["guardrail_decision"] == "pass":
        return "synthesize"
    elif state["guardrail_decision"] == "conditional":
        # Proceed but flag for officer review
        return "synthesize"
    else:  # fail
        return "safe_fallback"

workflow.add_conditional_edges("online_guardrail", guardrail_router)
workflow.add_edge("synthesize", "log_event")
workflow.add_edge("safe_fallback", "log_event")
workflow.add_edge("log_event", END)

from langgraph.checkpoint.postgres import PostgresSaver
app = workflow.compile(checkpointer=PostgresSaver.from_conn_string("postgresql://..."))

Step 4: Offline Evaluation Pipeline (Code Implementation)

This runs as a scheduled job, NOT inside the LangGraph. It consumes logged events and produces updated thresholds.

import pandas as pd
from sqlalchemy import create_engine
from datetime import datetime, timedelta

engine = create_engine("postgresql://...")

def run_offline_evaluation():
    """Weekly batch job: correlate online scores with actual outcomes."""
    
    # 1. Fetch matured recommendations (outcome observable after 90 days)
    query = """
        SELECT recommended_action, online_risk_score, actual_outcome, 
               recovery_amount_php, borrower_segment
        FROM repayment_recommendation_outcomes
        WHERE outcome_recorded_at >= NOW() - INTERVAL '90 days'
          AND actual_outcome IS NOT NULL
    """
    df = pd.read_sql(query, engine)
    
    # 2. Compute default rate by risk score bucket
    df["risk_bucket"] = pd.qcut(df["online_risk_score"], q=10, labels=False)
    bucket_stats = df.groupby("risk_bucket").agg(
        count=("actual_outcome", "count"),
        default_rate=("actual_outcome", lambda x: (x == "default").mean()),
        avg_recovery=("recovery_amount_php", "mean")
    ).reset_index()
    
    # 3. Detect miscalibration: if high-risk buckets have LOW default rates,
    #    online threshold is too conservative → relax it
    high_risk_low_default = bucket_stats[
        (bucket_stats["risk_bucket"] >= 7) & (bucket_stats["default_rate"] < 0.05)
    ]
    
    # 4. Fairness check: default rate disparity across segments
    segment_disparity = df.groupby("borrower_segment")["actual_outcome"].apply(
        lambda x: (x == "default").mean()
    )
    max_disparity = segment_disparity.max() - segment_disparity.min()
    
    # 5. Generate updated thresholds
    current_thresholds = load_thresholds_from_config()
    updated_thresholds = current_thresholds.copy()
    
    if len(high_risk_low_default) > 0:
        # Relax risk threshold by 5% (capped)
        updated_thresholds["min_risk_score_pass"] = max(
            0.3, current_thresholds["min_risk_score_pass"] - 0.05
        )
    
    if max_disparity > 0.08:  # >8pp disparity triggers alert
        send_alert_to_compliance(
            f"Fairness disparity detected: {max_disparity:.2%} across segments. "
            f"Segment breakdown:\n{segment_disparity.to_string()}"
        )
        # DO NOT auto-adjust; flag for human review
    
    # 6. Persist updated thresholds (loaded by online guardrail at next restart)
    save_thresholds_to_config(updated_thresholds)
    
    # 7. Log evaluation run for audit
    log_offline_eval_run(
        timestamp=datetime.utcnow(),
        samples=len(df),
        old_threshold=current_thresholds["min_risk_score_pass"],
        new_threshold=updated_thresholds["min_risk_score_pass"],
        fairness_disparity=max_disparity,
        calibration_adjustment=len(high_risk_low_default) > 0
    )
    
    return updated_thresholds

Part 4: Closing the Loop Safely

The critical architectural constraint: offline outputs NEVER directly modify online behavior without a validation gate.

# In your deployment pipeline, NOT in the LangGraph:
def deploy_updated_thresholds(new_thresholds):
    """Human-in-the-loop validation before online deployment."""
    
    # 1. Shadow test: run new thresholds against last 7 days of queries
    shadow_results = simulate_online_guardrail(new_thresholds, historical_queries)
    
    # 2. Check: would any previously-passed recommendations now fail?
    newly_blocked = shadow_results["newly_blocked_count"]
    if newly_blocked > 100:
        raise DeploymentBlockError(
            f"Threshold change would block {newly_blocked} additional recommendations. "
            "Requires manual review."
        )
    
    # 3. Check: does new threshold maintain minimum regulatory compliance?
    if new_thresholds["max_affordability_ratio"] > 0.40:
        raise DeploymentBlockError("Affordability threshold exceeds BSP limit.")
    
    # 4. Deploy atomically
    atomic_config_update(new_thresholds)
    notify_loan_operations(f"Online thresholds updated: {new_thresholds}")

Production Considerations for Microfinance

  1. Regulatory Immutability: Some thresholds (e.g., BSP affordability caps) are hard-coded constants, never adjusted by offline evaluation. Separate configurable vs. immutable thresholds in your config schema.

  2. Outcome Lag Handling: Repayment outcomes take months to observe. Use leading indicators (first payment after rescheduling, communication responsiveness) as early proxies in offline evaluation, validated against lagging default rates quarterly.

  3. Loan Officer Override Tracking: When officers override online recommendations, log BOTH the system recommendation and the override. Offline evaluation must analyze override outcomes separately—they're gold for calibrating trust boundaries.

  4. Regional Seasonality: Offline evaluation must be stratified by region and season. A threshold optimized for urban SME borrowers may catastrophically fail for agricultural borrowers during typhoon season. Maintain segment-specific threshold sets.

  5. Explainability for Regulators: Every online guardrail decision includes guardrail_reasoning. Store these immutably. During BSP examinations, you must demonstrate that every blocked/approved recommendation was traceable to specific, auditable criteria.

Conclusion

In microfinance repayment tracking, online and offline evaluation aren't alternatives—they're complementary halves of a responsible AI system. Online guardrails prevent immediate harm using the best available real-time knowledge. Offline evaluation ensures those guardrails evolve with portfolio reality, detecting miscalibration and unfairness before they compound. The LangGraph architecture makes this duality native: the online graph enforces constraints per-interaction, while structured event logging feeds the offline pipeline that continuously refines those constraints. Neither operates in isolation. Together, they form a system that is simultaneously safe today and better tomorrow—which is precisely what responsible microfinance demands.