Measuring Recommendation Quality in Enterprise Digital Wallets with LangGraph RAG

In digital wallet ecosystems, Click-Through Rate (CTR) is a dangerous north star. A user clicking on a "Buy Crypto" promotion might indicate curiosity, but if they abandon the transaction due to regulatory friction or insufficient balance, that click represents negative value. In fintech, recommendation quality must be measured by outcome fidelity, regulatory compliance, and semantic relevance—not just engagement. This article demonstrates how to build and evaluate a Digital Wallet Financial Advisor using LangGraph multi-agent RAG, where evaluation is baked directly into the orchestration layer.

The Real-Time Use Case: Personalized Financial Product Matching

Scenario: A digital wallet app with 2M+ users offers savings accounts, crypto trading, insurance micro-products, and merchant cashback. Users ask natural language questions like "How can I earn more on my idle PHP balance?" or "Is there a low-risk way to hedge against inflation?"

Why CTR Fails Here:

High-CTR items may violate suitability regulations for certain user risk profiles.
Users click exploratory content but convert on different products days later.
Regulatory penalties from mis-sold products dwarf ad revenue gains.

The Solution: A multi-agent system where a dedicated Evaluation Agent runs inline, measuring recommendation quality across four dimensions before any response reaches the user.

Part 1: The Four-Dimension Quality Framework

We replace CTR with a composite Recommendation Quality Score (RQS) computed per interaction:

Dimension	Metric	Why It Matters in Fintech
Semantic Fidelity	Embedding cosine similarity between query intent and recommendation rationale	Ensures the product actually answers the user's financial need
Regulatory Compliance	Binary pass/fail + confidence score from policy validator agent	Prevents mis-selling and regulatory fines
Business Outcome Alignment	Predicted conversion probability × expected lifetime value	Optimizes for revenue, not clicks
User Satisfaction Proxy	Implicit feedback signals (dwell time, save-for-later, explicit thumbs up/down)	Ground truth that replaces noisy CTR

These are not offline batch metrics. They are computed in real-time within the LangGraph state and stored for continuous evaluation.

Part 2: Data Representation for Wallet Context

User Financial Profile (ChromaDB Collection: `user_profiles`)

user_document = {
    "id": "user_wal_8847",
    "page_content": "Conservative investor. Primary goal: capital preservation. 
                     Has PHP 45,000 idle balance. Completed risk assessment Level 2. 
                     Previously declined crypto products twice. Enrolled in auto-save.",
    "metadata": {
        "type": "wallet_user",
        "risk_tolerance": "conservative",      # Key regulatory filter
        "kyc_level": "full",
        "idle_balance_php": 45000,
        "product_blacklist": ["crypto_spot", "leveraged_trading"],
        "jurisdiction": "PH",                  # Drives regulatory regime
        "last_risk_assessment_date": "2026-07-20"
    }
}

Financial Product Catalog (ChromaDB Collection: `financial_products`)

product_document = {
    "id": "prod_time_deposit_90d",
    "page_content": "90-day Time Deposit: 5.5% p.a. fixed rate. PDIC-insured up to PHP 500K. 
                     No early withdrawal penalty after 30 days. Minimum placement PHP 10,000.",
    "metadata": {
        "type": "financial_product",
        "category": "time_deposit",
        "risk_rating": "low",               # Must match user risk_tolerance
        "min_investment_php": 10000,
        "regulatory_tags": ["pdic_insured", "bsp_approved"],
        "eligible_kyc_levels": ["basic", "full"],
        "expected_conversion_rate": 0.34,    # Historical model output
        "avg_ltv_php": 2800                 # Business value signal
    }
}

Part 3: Multi-Agent Architecture with Inline Evaluation

Step 1: Define State with Evaluation Fields

from typing import Annotated, List, Dict, Optional
from typing_extensions import TypedDict
from langgraph.graph.message import add_messages
from langchain_core.documents import Document

class WalletRecState(TypedDict):
    messages: Annotated[List, add_messages]
    user_id: str
    user_profile: Optional[Document]
    
    # Retrieval & Ranking
    candidate_products: List[Document]
    ranked_recommendations: List[Dict]
    
    # === EVALUATION STATE (Beyond CTR) ===
    semantic_fidelity_score: float      # 0-1
    compliance_passed: bool
    business_alignment_score: float     # Normalized LTV×conversion
    overall_rqs: float                  # Composite score
    evaluation_reasoning: str           # Audit trail for regulators
    
    # Orchestration
    retry_count: int
    agent_trace: List[str]

Step 2: The Evaluation Agent (Core Innovation)

This agent runs before synthesis and determines whether recommendations are safe and relevant enough to show.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
import numpy as np

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

eval_prompt = ChatPromptTemplate.from_messages([
    ("system", """You are a Financial Recommendation Quality Auditor.
    Evaluate these recommendations against the user's profile and query.
    
    SCORING RUBRIC:
    - semantic_fidelity (0-1): Does each product directly address the stated financial need?
    - compliance_passed (bool): Do ALL products match user risk_tolerance, kyc_level, jurisdiction?
    - business_alignment (0-1): Weighted avg of (expected_conversion_rate × avg_ltv_php), normalized.
    
    Return JSON: {{
        "semantic_fidelity": float,
        "compliance_passed": bool,
        "business_alignment": float,
        "overall_rqs": float,  // 0.4*semantic + 0.4*compliance_binary + 0.2*business
        "reasoning": string
    }}"""),
    ("human", "Query: {query}\nUser Profile: {profile}\nRecommendations: {recs}")
])

def evaluation_agent_node(state: WalletRecState):
    chain = eval_prompt | llm.with_structured_output(dict)
    
    result = chain.invoke({
        "query": state["messages"][-1].content,
        "profile": state["user_profile"].page_content if state["user_profile"] else "Unknown",
        "recs": state["ranked_recommendations"]
    })
    
    rqs = result["overall_rqs"]
    
    return {
        "semantic_fidelity_score": result["semantic_fidelity"],
        "compliance_passed": result["compliance_passed"],
        "business_alignment_score": result["business_alignment"],
        "overall_rqs": rqs,
        "evaluation_reasoning": result["reasoning"],
        "agent_trace": state["agent_trace"] + [f"evaluator: RQS={rqs:.3f}, compliant={result['compliance_passed']}"]
    }

Step 3: Conditional Routing Based on Quality Threshold

from langgraph.graph import StateGraph, START, END

workflow = StateGraph(WalletRecState)

workflow.add_node("load_context", load_context_node)      # Loads user profile
workflow.add_node("retrieve", hybrid_retriever_node)       # Semantic + metadata filter
workflow.add_node("compliance_check", compliance_node)     # Hard regulatory gate
workflow.add_node("rank", business_ranker_node)            # LTV-weighted sorting
workflow.add_node("evaluate", evaluation_agent_node)       # ← QUALITY GATE
workflow.add_node("synthesize", synthesizer_node)
workflow.add_node("safe_fallback", fallback_node)          # Generic safe advice

workflow.add_edge(START, "load_context")
workflow.add_edge("load_context", "retrieve")
workflow.add_edge("retrieve", "compliance_check")
workflow.add_edge("compliance_check", "rank")
workflow.add_edge("rank", "evaluate")

# KEY: Route based on RQS, not retrieval count
def quality_gate(state: WalletRecState):
    if not state["compliance_passed"]:
        return "safe_fallback"  # Never show non-compliant recs
    if state["overall_rqs"] >= 0.65:
        return "synthesize"
    if state["retry_count"] < 2:
        return "retrieve"       # Retry with relaxed semantic threshold
    return "safe_fallback"      # Graceful degradation

workflow.add_conditional_edges("evaluate", quality_gate)
workflow.add_edge("synthesize", END)
workflow.add_edge("safe_fallback", END)

from langgraph.checkpoint.postgres import PostgresSaver
app = workflow.compile(checkpointer=PostgresSaver.from_conn_string("postgresql://..."))

Step 4: Execution & Metric Persistence

config = {"configurable": {"thread_id": "session_wal8847_20260812"}}

result = app.invoke({
    "messages": [("human", "What can I do with my extra PHP 45K to beat inflation safely?")],
    "user_id": "user_wal_8847",
    "retry_count": 0,
    "agent_trace": [],
    # ... initialize all state fields
}, config=config)

# Persist evaluation metrics to your analytics warehouse
log_evaluation_event(
    user_id=result["user_id"],
    thread_id=config["configurable"]["thread_id"],
    rqs=result["overall_rqs"],
    semantic=result["semantic_fidelity_score"],
    compliance=result["compliance_passed"],
    business=result["business_alignment_score"],
    reasoning=result["evaluation_reasoning"],
    timestamp=datetime.utcnow()
)

Part 4: Closing the Feedback Loop

Real-time RQS is only half the equation. You must correlate it with delayed outcomes:

# Offline job: Correlate RQS with actual conversions (7-day window)
SELECT 
    rqs_bucket,
    COUNT(*) as impressions,
    SUM(conversion_flag) as conversions,
    AVG(ltv_realized_php) as avg_ltv
FROM recommendation_events
WHERE event_date >= CURRENT_DATE - INTERVAL '7 days'
GROUP BY NTILE(10) OVER (ORDER BY overall_rqs) as rqs_bucket;

If high-RQS buckets don't correlate with conversions, your evaluation rubric needs recalibration. This is how you replace CTR with a validated business metric.

Production Considerations for Fintech

Regulatory Audit Trail: Every evaluation_reasoning string must be immutable and retained per jurisdiction requirements (e.g., BSP, MAS, SEC). Store in append-only storage.
Model Risk Management: The Evaluation Agent is a model. Validate its scoring against human-labeled golden datasets quarterly. Document its methodology for regulators.
Latency Budget: Inline evaluation adds ~800ms. Pre-compute compliance flags at ingestion time. Cache user profiles aggressively. Use streaming synthesis so perceived latency stays low.
Fallback Safety: The safe_fallback node must return pre-approved, regulator-reviewed generic content. Never let the LLM freestyle when RQS fails.
A/B Test RQS Thresholds: The 0.65 gate is a hypothesis. Run controlled experiments comparing threshold values against actual NPS and conversion rates.

Conclusion

In digital wallets, recommendation quality isn't a post-hoc analytics problem—it's a real-time safety and business constraint. By embedding a dedicated Evaluation Agent within your LangGraph orchestration, computing multi-dimensional RQS before synthesis, and routing based on quality gates rather than retrieval success, you build systems that optimize for what actually matters: compliant, valuable financial outcomes. CTR tells you what users clicked. RQS tells you whether you should have shown it at all. In regulated fintech, that distinction is everything.