The LLM Selection Challenge in Regulated Industries

Introduction: The LLM Selection Challenge in Regulated Industries

When building enterprise AI for banking, the question is never just "Which model is smartest?" It is "Which model is most reliable, compliant, and cost-effective for this specific workflow?" Linguistic fluency is table stakes; domain correctness is the differentiator.

Comparing LLMs for a regulated use case requires a structured evaluation process that goes beyond generic benchmarks. You must test models against your actual data, your specific compliance rules, and your latency budgets. A model that scores higher on MMLU might still fail at interpreting your bank’s internal credit policy correctly.

This article details a rigorous LLM comparison framework and demonstrates it through a complete implementation of "ReguCheck," an enterprise-grade multi-agent system built with LangGraph, RAG, memory, and state management. This system doesn’t just answer questions—it validates them, remembers context, and provides auditable reasoning traces.

The Evaluation Framework: Comparing Models for Domain Correctness

Before writing code, establish a comparison methodology. Here is the process used to evaluate LLMs for bank policy tasks:

1. Define Task-Specific Metrics

Generic benchmarks are irrelevant. Define metrics tied to business outcomes:

Factual Groundedness: % of claims directly supported by retrieved policy chunks.
Compliance Adherence: % of responses passing automated regulatory checks.
Reasoning Completeness: Does the CoT trace cover all required policy clauses?
Latency (TTFT + TPS): Time-to-first-token and tokens-per-second under load.
Cost per Verified Answer: Total API cost divided by compliant responses (not total responses).

2. Build a Golden Test Set

Create 100–200 annotated query-answer pairs from real bank operations:

Simple factual lookups ("What is the max tenure for home loans?")
Multi-hop reasoning ("Can a self-employed applicant with a 720 CIBIL score get a gold loan?")
Edge cases with conflicting policies
Adversarial queries designed to trigger hallucinations

Each entry includes: expected answer, required source documents, and compliance flags.

3. Run Controlled Experiments

Use the same RAG pipeline, same prompts, same retrieval top-k across all candidate models. Only swap the LLM. Measure:

Retrieval recall/precision (constant across models)
Answer correctness (variable)
Compliance pass rate (variable)
Token usage and latency (variable)

4. Evaluate Beyond the Final Answer

Inspect intermediate outputs:

Are reasoning steps logically sound even if the final answer is wrong?
Does the model correctly identify when information is missing?
How well does it follow structured output formats?

5. Make a Weighted Decision

Create a scoring matrix. In banking, compliance adherence and groundedness typically carry 3x weight over fluency or speed. The "best" model is the one that maximizes verified correct answers within budget—not the one with the highest benchmark score.

Real-Time Use Case: "ReguCheck" – Multi-Agent Bank Policy Compliance System

Scenario: GlobalTrust Bank needs an internal assistant for branch managers handling loan eligibility queries. Policies span hundreds of PDFs updated monthly. Answers must be RBI-compliant, cite sources, and maintain conversation context.

Business Problem: Standard chatbots hallucinate interest rates, miss conditional clauses, and provide no audit trail. Managers cannot trust AI outputs for customer-facing decisions.

AI Solution: A LangGraph-based multi-agent system where:

A Router Agent classifies query intent
A Retrieval Agent fetches relevant policy chunks
An Analyst Agent reasons step-by-step using CoT
A Compliance Agent validates the draft against source text
State carries all intermediate artifacts for auditing
Memory preserves session context for follow-ups

This architecture makes LLM comparison meaningful because you can measure each model's performance within the validation loop, not in isolation.

Architecture Overview: Enterprise LangGraph + RAG with Memory & State

Key Architectural Decisions:

LangGraph over Chains: Enables conditional routing (compliance rejection loops), persistent state, and explicit node boundaries for evaluation.
Separate Compliance Agent: Decouples generation from validation, allowing independent model selection (e.g., cheaper model for retrieval, stronger model for compliance).
Structured State: Pydantic-enforced state schema ensures type safety and makes every intermediate artifact inspectable for evaluation.
Session Memory: Conversation history stored separately from working state enables multi-turn reasoning without polluting the current task context.

Agent Design: Specialized Roles for Accuracy and Safety

Agent	Responsibility	Why Separate?
Router	Classify query domain (Loan/KYC/Fraud/General)	Prevents irrelevant retrieval; enables domain-specific prompting
Retriever	Vector search + metadata filtering	Isolation allows swapping embedding models independently
Analyst	CoT reasoning over retrieved chunks	Focused prompt improves reasoning quality; easier to evaluate
Compliance	Validate draft against source text	Acts as adversarial critic; can use different model than analyst
Responder	Format output, cite sources, update memory	Ensures consistent UX regardless of upstream model changes

Why not one agent? Monolithic prompts conflate retrieval, reasoning, and validation. When evaluating models, you cannot tell if failures stem from poor retrieval, weak reasoning, or inadequate validation. Separation enables granular model comparison per task.

RAG Pipeline: Grounding Responses in Bank Policy Documents

Document Ingestion

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = PyPDFLoader("bank_policies/home_loan_v3.pdf")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=80,
    separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)

Embeddings & Vector Store

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma.from_documents(
    chunks, embeddings,
    persist_directory="./bank_policy_db",
    collection_metadata={"hnsw:space": "cosine"}
)

Retrieval with Metadata Filtering

def retrieve(query: str, domain: str, k: int = 4):
    return vector_store.similarity_search_with_score(
        query, k=k,
        filter={"domain": domain}  # Pre-filter by routed domain
    )

Evaluation Note: Keep retrieval constant during LLM comparison. If retrieval quality varies, you’re testing the retriever, not the LLM.

LangGraph Workflow: Orchestrating Stateful Multi-Agent Reasoning

State Definition

from typing import Annotated, List, Optional, Dictfrom pydantic import BaseModel, Field
import operator

class ReguCheckState(BaseModel):
    messages: Annotated[List[dict], operator.add] = Field(default_factory=list)
    user_query: str = ""
    domain: Optional[str] = None
    retrieved_chunks: List[Dict[str, str]] = Field(default_factory=list)
    reasoning_trace: List[str] = Field(default_factory=list)
    draft_answer: Optional[str] = None
    compliance_status: Optional[str] = None  # APPROVED / REJECTED
    compliance_feedback: Optional[str] = None
    final_answer: Optional[str] = None
    sources: List[str] = Field(default_factory=list)

Conditional Routing

def route_after_compliance(state: ReguCheckState) -> str:
    if state.compliance_status == "REJECTED":
        return "analyst"  # Loop back for revision
    return "responder"

This conditional edge is critical for evaluation: it measures how often each model produces compliant-first drafts vs. requiring correction loops.

Memory & State Management: Session Context and Audit Trails

Working State (ReguCheckState): Ephemeral per-query. Contains retrieval results, reasoning traces, compliance status. Used for intra-graph communication and audit logging.
Session Memory (messages): Persistent across turns within a session. Stored in Redis or PostgreSQL. Enables follow-up questions like "What about the prepayment penalty for that loan?"
Audit Trail: Every state transition is logged with timestamp, agent name, model used, and token count. This is non-negotiable for regulatory compliance and model evaluation.

Complete Code Implementation: End-to-End Python Solution

Project Structure

regucheck/
├── main.py              # FastAPI app + graph compilation
├── state.py             # ReguCheckState definition
├── agents/
│   ├── router.py
│   ├── retriever.py
│   ├── analyst.py
│   ├── compliance.py
│   └── responder.py
├── rag/
│   ├── ingestion.py
│   └── retrieval.py
├── memory/
│   └── session_store.py
├── eval/
│   └── compare_models.py
└── config.py

Core Graph Compilation (`main.py`)

from langgraph.graph import StateGraph, START, END
from state import ReguCheckState
from agents.router import route_query
from agents.retriever import retrieve_policies
from agents.analyst import analyze_policy
from agents.compliance import check_compliance
from agents.responder import format_response

workflow = StateGraph(ReguCheckState)

workflow.add_node("router", route_query)
workflow.add_node("retriever", retrieve_policies)
workflow.add_node("analyst", analyze_policy)
workflow.add_node("compliance", check_compliance)
workflow.add_node("responder", format_response)

workflow.add_edge(START, "router")
workflow.add_edge("router", "retriever")
workflow.add_edge("retriever", "analyst")
workflow.add_edge("analyst", "compliance")
workflow.add_conditional_edges("compliance", 
    lambda s: "analyst" if s.compliance_status == "REJECTED" else "responder",
    {"analyst": "analyst", "responder": "responder"}
)
workflow.add_edge("responder", END)

app = workflow.compile()

Model Comparison Script (`eval/compare_models.py`)

import json
from langchain_openai import ChatOpenAI

MODELS = ["gpt-4o-mini", "gpt-4o", "claude-sonnet-4-20250514"]
TEST_QUERIES = json.load(open("golden_test_set.json"))

results = {}
for model_name in MODELS:
    llm = ChatOpenAI(model=model_name, temperature=0.1)
    # Inject llm into agents via dependency injection or global config
    set_active_llm(llm)
    
    metrics = {"groundedness": [], "compliance_pass": [], "latency_ms": [], "tokens": []}
    for test_case in TEST_QUERIES:
        state = ReguCheckState(user_query=test_case["query"])
        result = app.invoke(state)
        
        metrics["groundedness"].append(evaluate_groundedness(result, test_case))
        metrics["compliance_pass"].append(result.compliance_status == "APPROVED")
        metrics["latency_ms"].append(result.metadata.get("latency_ms", 0))
        metrics["tokens"].append(result.metadata.get("total_tokens", 0))
    
    results[model_name] = {k: sum(v)/len(v) if isinstance(v[0], (int,float,bool)) else v 
                           for k, v in metrics.items()}

print(json.dumps(results, indent=2))

Execution Walkthrough: Tracing a Query Through the Graph

Query: "I have a CIBIL score of 720 and am self-employed. Can I get a home loan with 5% down payment?"

Router: Classifies as home_loan. Sets state.domain = "home_loan".
Retriever: Fetches 4 chunks matching "self-employed home loan CIBIL down payment". Updates state.retrieved_chunks.
Analyst: Generates CoT: "Step 1: Self-employed eligible per Section 3.2. Step 2: CIBIL 720 falls in 700-749 bracket → 9.5% rate + 10% min down payment per Section 4.1. Step 3: 5% < 10% → Not eligible." Drafts answer. Updates state.draft_answer and state.reasoning_trace.
Compliance: Validates draft against Section 4.1. Confirms 10% minimum is correctly cited. Returns APPROVED. Updates state.compliance_status.
Responder: Formats answer with source citations. Appends to state.messages. Sets state.final_answer.
Output: Returns verified answer with full audit trail.

If Compliance had returned REJECTED, the graph would loop back to Analyst with feedback, and the retry count would be logged for model evaluation.

Enterprise Considerations: Security, Compliance, and Observability

PII Redaction: Pre-process queries to mask account numbers, Aadhaar, PAN before LLM calls.
Authentication: OAuth2/JWT validation on API endpoints; role-based access to sensitive policies.
Audit Logging: Log every state transition to append-only storage (S3/Azure Blob with WORM). Include model version, prompt hash, and token counts.
Guardrails: Input/output filtering for prompt injection and prohibited content.
Observability: Trace each graph execution with LangSmith/Langfuse. Correlate latency, token usage, and compliance outcomes.
Human-in-the-Loop: Route low-confidence or repeatedly rejected answers to human reviewers.
Data Privacy: Deploy embeddings and vector DB within VPC. Use Azure OpenAI or private LLM deployments for sensitive policies.

Testing Strategy: Evaluating Agents, RAG, and Model Performance

Component	Test Type	Metric	Tool
Retriever	Retrieval Eval	Recall@K, Precision@K	RAGAS, Aries
Analyst	Reasoning Eval	CoT completeness, logical validity	Custom rubric + LLM-as-judge
Compliance	Validation Eval	False positive/negative rate	Golden test set
Full Graph	E2E Eval	Groundedness, compliance pass rate, latency	compare_models.py
Memory	Session Eval	Follow-up accuracy, context retention	Multi-turn test suite

Critical: Always test with production-like data volume and concurrency. Latency and cost profiles change dramatically under load.

Common Pitfalls & Solutions

Problem	Root Cause	Solution
Wrong retrieval	Poor chunking or missing metadata	Hybrid search + domain-tagged metadata
Compliance false rejections	Overly strict validator prompt	Calibrate with false positive analysis
Infinite compliance loops	Analyst ignores feedback	Max retry limit + escalation to human
Lost session context	Memory not persisted between requests	Redis/PostgreSQL-backed session store
High latency	Sequential agent calls	Parallel retrieval + streaming responses
Model works in eval but fails in prod	Distribution shift in live queries	Continuous monitoring + golden set expansion

Future Enhancements

Hybrid Search: Combine BM25 keyword search with vector similarity for policy clause lookup.
Re-ranking: Cross-encoder re-ranker after initial retrieval to improve precision.
Live Data Integration: MCP tools to fetch real-time account/balance data alongside static policies.
Model Routing: Use lightweight classifier to route simple queries to cheap models, complex ones to premium models.
Production Monitoring: Drift detection on compliance pass rates and latency percentiles.
Fine-tuning: Distill top-performing model’s reasoning traces into smaller, faster model for high-volume queries.

Conclusion: Building Trustworthy AI Through Rigorous Evaluation

Comparing LLMs for enterprise banking is not about finding the "smartest" model. It is about finding the model that delivers the highest rate of verified, compliant, auditable answers within operational constraints. This requires:

A task-specific evaluation framework with golden test sets and domain-relevant metrics.
A multi-agent architecture that separates concerns and enables granular measurement.
Stateful orchestration via LangGraph that makes every reasoning step inspectable.
Continuous validation through compliance agents and audit trails.

The ReguCheck system demonstrated here provides both the infrastructure for production deployment and the instrumentation needed for rigorous model comparison. In regulated industries, trust is not given—it is engineered, measured, and continuously validated.