Dual-Track Evaluation for Microfinance Repayment Recommendations: Online Safety Gates & Offline Policy Optimization with LangGraph

Step 2: Online Guardrail Agent (Real-Time Safety Gate)

This agent enforces hard constraints derived from offline-optimized thresholds loaded at graph initialization.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
import json

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

# Thresholds loaded from offline evaluation pipeline at startup
# These are UPDATED periodically, not hardcoded
ONLINE_THRESHOLDS = load_thresholds_from_config()  
# e.g., {"max_affordability_ratio": 0.40, "min_risk_score_pass": 0.65, 
#         "required_docs_agricultural": ["barangay_cert"]}

guardrail_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a Microfinance Repayment Compliance Auditor.
    Evaluate this proposed rescheduling against BSP regulations and internal policy.
    
    THRESHOLDS (updated via offline evaluation):
    - Max affordability ratio: {max_affordability}
    - Min risk score to pass: {min_risk_score}
    - Required documentation for {segment}: {required_docs}
    
    BORROWER STATE: {profile}
    APPLICABLE POLICIES: {policies}
    PROPOSED ACTION: {proposed}
    
    Return JSON: {{
        "affordability_ratio": float or null,
        "documentation_complete": bool,
        "regulatory_compliant": bool,
        "online_risk_score": float,
        "decision": "pass" | "fail" | "conditional",
        "reasoning": string
    }}"""),
    ("human", "{query}")
])

def online_guardrail_node(state: RepaymentRecState):
    chain = guardrail_prompt | llm.with_structured_output(dict)
    
    result = chain.invoke({
        "max_affordability": ONLINE_THRESHOLDS["max_affordability_ratio"],
        "min_risk_score": ONLINE_THRESHOLDS["min_risk_score_pass"],
        "segment": state["borrower_profile"].metadata["segment"] if state["borrower_profile"] else "unknown",
        "required_docs": ONLINE_THRESHOLDS.get(
            f"required_docs_{state['borrower_profile'].metadata.get('segment', 'general')}", []
        ),
        "profile": state["borrower_profile"].page_content if state["borrower_profile"] else "No profile",
        "policies": "\n---\n".join(p.page_content for p in state["relevant_policies"]),
        "proposed": state["messages"][-1].content,
        "query": state["messages"][-1].content
    })
    
    return {
        "affordability_ratio": result["affordability_ratio"],
        "documentation_complete": result["documentation_complete"],
        "regulatory_compliant": result["regulatory_compliant"],
        "online_risk_score": result["online_risk_score"],
        "guardrail_decision": result["decision"],
        "guardrail_reasoning": result["reasoning"],
        "agent_trace": state["agent_trace"] + [
            f"guardrail: decision={result['decision']}, risk={result['online_risk_score']:.3f}"
        ]
    }

Step 3: Conditional Routing Based on Online Evaluation

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(RepaymentRecState)

workflow.add_node("load_borrower", load_borrower_node)
workflow.add_node("retrieve_policies", policy_retriever_node)
workflow.add_node("online_guardrail", online_guardrail_node)
workflow.add_node("synthesize", recommendation_synthesizer_node)
workflow.add_node("safe_fallback", regulatory_safe_fallback_node)
workflow.add_node("log_event", log_recommendation_event_node)  # ← Feeds offline pipeline

workflow.add_edge(START, "load_borrower")
workflow.add_edge("load_borrower", "retrieve_policies")
workflow.add_edge("retrieve_policies", "online_guardrail")

def guardrail_router(state: RepaymentRecState):
    if state["guardrail_decision"] == "pass":
        return "synthesize"
    elif state["guardrail_decision"] == "conditional":
        # Proceed but flag for officer review
        return "synthesize"
    else:  # fail
        return "safe_fallback"

workflow.add_conditional_edges("online_guardrail", guardrail_router)
workflow.add_edge("synthesize", "log_event")
workflow.add_edge("safe_fallback", "log_event")
workflow.add_edge("log_event", END)

from langgraph.checkpoint.postgres import PostgresSaver
app = workflow.compile(checkpointer=PostgresSaver.from_conn_string("postgresql://..."))

Step 4: Offline Evaluation Pipeline (Code Implementation)

This runs as a scheduled job, NOT inside the LangGraph. It consumes logged events and produces updated thresholds.

import pandas as pd
from sqlalchemy import create_engine
from datetime import datetime, timedelta

engine = create_engine("postgresql://...")

def run_offline_evaluation():
    """Weekly batch job: correlate online scores with actual outcomes."""
    
    # 1. Fetch matured recommendations (outcome observable after 90 days)
    query = """
        SELECT recommended_action, online_risk_score, actual_outcome, 
               recovery_amount_php, borrower_segment
        FROM repayment_recommendation_outcomes
        WHERE outcome_recorded_at >= NOW() - INTERVAL '90 days'
          AND actual_outcome IS NOT NULL
    """
    df = pd.read_sql(query, engine)
    
    # 2. Compute default rate by risk score bucket
    df["risk_bucket"] = pd.qcut(df["online_risk_score"], q=10, labels=False)
    bucket_stats = df.groupby("risk_bucket").agg(
        count=("actual_outcome", "count"),
        default_rate=("actual_outcome", lambda x: (x == "default").mean()),
        avg_recovery=("recovery_amount_php", "mean")
    ).reset_index()
    
    # 3. Detect miscalibration: if high-risk buckets have LOW default rates,
    #    online threshold is too conservative → relax it
    high_risk_low_default = bucket_stats[
        (bucket_stats["risk_bucket"] >= 7) & (bucket_stats["default_rate"] < 0.05)
    ]
    
    # 4. Fairness check: default rate disparity across segments
    segment_disparity = df.groupby("borrower_segment")["actual_outcome"].apply(
        lambda x: (x == "default").mean()
    )
    max_disparity = segment_disparity.max() - segment_disparity.min()
    
    # 5. Generate updated thresholds
    current_thresholds = load_thresholds_from_config()
    updated_thresholds = current_thresholds.copy()
    
    if len(high_risk_low_default) > 0:
        # Relax risk threshold by 5% (capped)
        updated_thresholds["min_risk_score_pass"] = max(
            0.3, current_thresholds["min_risk_score_pass"] - 0.05
        )
    
    if max_disparity > 0.08:  # >8pp disparity triggers alert
        send_alert_to_compliance(
            f"Fairness disparity detected: {max_disparity:.2%} across segments. "
            f"Segment breakdown:\n{segment_disparity.to_string()}"
        )
        # DO NOT auto-adjust; flag for human review
    
    # 6. Persist updated thresholds (loaded by online guardrail at next restart)
    save_thresholds_to_config(updated_thresholds)
    
    # 7. Log evaluation run for audit
    log_offline_eval_run(
        timestamp=datetime.utcnow(),
        samples=len(df),
        old_threshold=current_thresholds["min_risk_score_pass"],
        new_threshold=updated_thresholds["min_risk_score_pass"],
        fairness_disparity=max_disparity,
        calibration_adjustment=len(high_risk_low_default) > 0
    )
    
    return updated_thresholds

Part 4: Closing the Loop Safely

The critical architectural constraint: offline outputs NEVER directly modify online behavior without a validation gate.

# In your deployment pipeline, NOT in the LangGraph:
def deploy_updated_thresholds(new_thresholds):
    """Human-in-the-loop validation before online deployment."""
    
    # 1. Shadow test: run new thresholds against last 7 days of queries
    shadow_results = simulate_online_guardrail(new_thresholds, historical_queries)
    
    # 2. Check: would any previously-passed recommendations now fail?
    newly_blocked = shadow_results["newly_blocked_count"]
    if newly_blocked > 100:
        raise DeploymentBlockError(
            f"Threshold change would block {newly_blocked} additional recommendations. "
            "Requires manual review."
        )
    
    # 3. Check: does new threshold maintain minimum regulatory compliance?
    if new_thresholds["max_affordability_ratio"] > 0.40:
        raise DeploymentBlockError("Affordability threshold exceeds BSP limit.")
    
    # 4. Deploy atomically
    atomic_config_update(new_thresholds)
    notify_loan_operations(f"Online thresholds updated: {new_thresholds}")

Production Considerations for Microfinance

1. Regulatory Immutability: Some thresholds (e.g., BSP affordability caps) are hard-coded constants, never adjusted by offline evaluation. Separate configurable vs. immutable thresholds in your config schema.

2. Outcome Lag Handling: Repayment outcomes take months to observe. Use leading indicators (first payment after rescheduling, communication responsiveness) as early proxies in offline evaluation, validated against lagging default rates quarterly.

3. Loan Officer Override Tracking: When officers override online recommendations, log BOTH the system recommendation and the override. Offline evaluation must analyze override outcomes separately—they're gold for calibrating trust boundaries.

4. Regional Seasonality: Offline evaluation must be stratified by region and season. A threshold optimized for urban SME borrowers may catastrophically fail for agricultural borrowers during typhoon season. Maintain segment-specific threshold sets.

5. Explainability for Regulators: Every online guardrail decision includes guardrail_reasoning. Store these immutably. During BSP examinations, you must demonstrate that every blocked/approved recommendation was traceable to specific, auditable criteria.

Conclusion

In microfinance repayment tracking, online and offline evaluation aren't alternatives—they're complementary halves of a responsible AI system. Online guardrails prevent immediate harm using the best available real-time knowledge. Offline evaluation ensures those guardrails evolve with portfolio reality, detecting miscalibration and unfairness before they compound. The LangGraph architecture makes this duality native: the online graph enforces constraints per-interaction, while structured event logging feeds the offline pipeline that continuously refines those constraints. Neither operates in isolation. Together, they form a system that is simultaneously safe today and better tomorrow—which is precisely what responsible microfinance demands.