Introduction: The LLM Selection Challenge in Regulated Industries

When building enterprise AI for banking, the question is never just "Which model is smartest?" It is "Which model is most reliable, compliant, and cost-effective for this specific workflow?" Linguistic fluency is table stakes; domain correctness is the differentiator.

Comparing LLMs for a regulated use case requires a structured evaluation process that goes beyond generic benchmarks. You must test models against your actual data, your specific compliance rules, and your latency budgets. A model that scores higher on MMLU might still fail at interpreting your bank’s internal credit policy correctly.

This article details a rigorous LLM comparison framework and demonstrates it through a complete implementation of "ReguCheck," an enterprise-grade multi-agent system built with LangGraph, RAG, memory, and state management. This system doesn’t just answer questions—it validates them, remembers context, and provides auditable reasoning traces.

The Evaluation Framework: Comparing Models for Domain Correctness

Before writing code, establish a comparison methodology. Here is the process used to evaluate LLMs for bank policy tasks:

1. Define Task-Specific Metrics

Generic benchmarks are irrelevant. Define metrics tied to business outcomes:

2. Build a Golden Test Set

Create 100–200 annotated query-answer pairs from real bank operations:

Each entry includes: expected answer, required source documents, and compliance flags.

3. Run Controlled Experiments

Use the same RAG pipeline, same prompts, same retrieval top-k across all candidate models. Only swap the LLM. Measure:

4. Evaluate Beyond the Final Answer

Inspect intermediate outputs:

5. Make a Weighted Decision

Create a scoring matrix. In banking, compliance adherence and groundedness typically carry 3x weight over fluency or speed. The "best" model is the one that maximizes verified correct answers within budget—not the one with the highest benchmark score.

Real-Time Use Case: "ReguCheck" – Multi-Agent Bank Policy Compliance System

Scenario: GlobalTrust Bank needs an internal assistant for branch managers handling loan eligibility queries. Policies span hundreds of PDFs updated monthly. Answers must be RBI-compliant, cite sources, and maintain conversation context.

Business Problem: Standard chatbots hallucinate interest rates, miss conditional clauses, and provide no audit trail. Managers cannot trust AI outputs for customer-facing decisions.

AI Solution: A LangGraph-based multi-agent system where:

This architecture makes LLM comparison meaningful because you can measure each model's performance within the validation loop, not in isolation.

Architecture Overview: Enterprise LangGraph + RAG with Memory & State

465

Key Architectural Decisions:

Agent Design: Specialized Roles for Accuracy and Safety

AgentResponsibilityWhy Separate?
RouterClassify query domain (Loan/KYC/Fraud/General)Prevents irrelevant retrieval; enables domain-specific prompting
RetrieverVector search + metadata filteringIsolation allows swapping embedding models independently
AnalystCoT reasoning over retrieved chunksFocused prompt improves reasoning quality; easier to evaluate
ComplianceValidate draft against source textActs as adversarial critic; can use different model than analyst
ResponderFormat output, cite sources, update memoryEnsures consistent UX regardless of upstream model changes

Why not one agent? Monolithic prompts conflate retrieval, reasoning, and validation. When evaluating models, you cannot tell if failures stem from poor retrieval, weak reasoning, or inadequate validation. Separation enables granular model comparison per task.

RAG Pipeline: Grounding Responses in Bank Policy Documents

Document Ingestion

from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter

loader = PyPDFLoader("bank_policies/home_loan_v3.pdf")
docs = loader.load()

splitter = RecursiveCharacterTextSplitter(
    chunk_size=600,
    chunk_overlap=80,
    separators=["\n\n", "\n", ". ", " "]
)
chunks = splitter.split_documents(docs)

Embeddings & Vector Store

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vector_store = Chroma.from_documents(
    chunks, embeddings,
    persist_directory="./bank_policy_db",
    collection_metadata={"hnsw:space": "cosine"}
)

Retrieval with Metadata Filtering

def retrieve(query: str, domain: str, k: int = 4):
    return vector_store.similarity_search_with_score(
        query, k=k,
        filter={"domain": domain}  # Pre-filter by routed domain
    )

Evaluation Note: Keep retrieval constant during LLM comparison. If retrieval quality varies, you’re testing the retriever, not the LLM.

LangGraph Workflow: Orchestrating Stateful Multi-Agent Reasoning

State Definition

from typing import Annotated, List, Optional, Dictfrom pydantic import BaseModel, Field
import operator

class ReguCheckState(BaseModel):
    messages: Annotated[List[dict], operator.add] = Field(default_factory=list)
    user_query: str = ""
    domain: Optional[str] = None
    retrieved_chunks: List[Dict[str, str]] = Field(default_factory=list)
    reasoning_trace: List[str] = Field(default_factory=list)
    draft_answer: Optional[str] = None
    compliance_status: Optional[str] = None  # APPROVED / REJECTED
    compliance_feedback: Optional[str] = None
    final_answer: Optional[str] = None
    sources: List[str] = Field(default_factory=list)

Conditional Routing

def route_after_compliance(state: ReguCheckState) -> str:
    if state.compliance_status == "REJECTED":
        return "analyst"  # Loop back for revision
    return "responder"

This conditional edge is critical for evaluation: it measures how often each model produces compliant-first drafts vs. requiring correction loops.

Memory & State Management: Session Context and Audit Trails

Complete Code Implementation: End-to-End Python Solution

Project Structure

regucheck/
├── main.py              # FastAPI app + graph compilation
├── state.py             # ReguCheckState definition
├── agents/
│   ├── router.py
│   ├── retriever.py
│   ├── analyst.py
│   ├── compliance.py
│   └── responder.py
├── rag/
│   ├── ingestion.py
│   └── retrieval.py
├── memory/
│   └── session_store.py
├── eval/
│   └── compare_models.py
└── config.py

Core Graph Compilation (main.py)

from langgraph.graph import StateGraph, START, END
from state import ReguCheckState
from agents.router import route_query
from agents.retriever import retrieve_policies
from agents.analyst import analyze_policy
from agents.compliance import check_compliance
from agents.responder import format_response

workflow = StateGraph(ReguCheckState)

workflow.add_node("router", route_query)
workflow.add_node("retriever", retrieve_policies)
workflow.add_node("analyst", analyze_policy)
workflow.add_node("compliance", check_compliance)
workflow.add_node("responder", format_response)

workflow.add_edge(START, "router")
workflow.add_edge("router", "retriever")
workflow.add_edge("retriever", "analyst")
workflow.add_edge("analyst", "compliance")
workflow.add_conditional_edges("compliance", 
    lambda s: "analyst" if s.compliance_status == "REJECTED" else "responder",
    {"analyst": "analyst", "responder": "responder"}
)
workflow.add_edge("responder", END)

app = workflow.compile()

Model Comparison Script (eval/compare_models.py)

import json
from langchain_openai import ChatOpenAI

MODELS = ["gpt-4o-mini", "gpt-4o", "claude-sonnet-4-20250514"]
TEST_QUERIES = json.load(open("golden_test_set.json"))

results = {}
for model_name in MODELS:
    llm = ChatOpenAI(model=model_name, temperature=0.1)
    # Inject llm into agents via dependency injection or global config
    set_active_llm(llm)
    
    metrics = {"groundedness": [], "compliance_pass": [], "latency_ms": [], "tokens": []}
    for test_case in TEST_QUERIES:
        state = ReguCheckState(user_query=test_case["query"])
        result = app.invoke(state)
        
        metrics["groundedness"].append(evaluate_groundedness(result, test_case))
        metrics["compliance_pass"].append(result.compliance_status == "APPROVED")
        metrics["latency_ms"].append(result.metadata.get("latency_ms", 0))
        metrics["tokens"].append(result.metadata.get("total_tokens", 0))
    
    results[model_name] = {k: sum(v)/len(v) if isinstance(v[0], (int,float,bool)) else v 
                           for k, v in metrics.items()}

print(json.dumps(results, indent=2))

Execution Walkthrough: Tracing a Query Through the Graph

Query: "I have a CIBIL score of 720 and am self-employed. Can I get a home loan with 5% down payment?"

  1. Router: Classifies as home_loan. Sets state.domain = "home_loan".

  2. Retriever: Fetches 4 chunks matching "self-employed home loan CIBIL down payment". Updates state.retrieved_chunks.

  3. Analyst: Generates CoT: "Step 1: Self-employed eligible per Section 3.2. Step 2: CIBIL 720 falls in 700-749 bracket → 9.5% rate + 10% min down payment per Section 4.1. Step 3: 5% < 10% → Not eligible." Drafts answer. Updates state.draft_answer and state.reasoning_trace.

  4. Compliance: Validates draft against Section 4.1. Confirms 10% minimum is correctly cited. Returns APPROVED. Updates state.compliance_status.

  5. Responder: Formats answer with source citations. Appends to state.messages. Sets state.final_answer.

  6. Output: Returns verified answer with full audit trail.

If Compliance had returned REJECTED, the graph would loop back to Analyst with feedback, and the retry count would be logged for model evaluation.

Enterprise Considerations: Security, Compliance, and Observability

Testing Strategy: Evaluating Agents, RAG, and Model Performance

ComponentTest TypeMetricTool
RetrieverRetrieval EvalRecall@K, Precision@KRAGAS, Aries
AnalystReasoning EvalCoT completeness, logical validityCustom rubric + LLM-as-judge
ComplianceValidation EvalFalse positive/negative rateGolden test set
Full GraphE2E EvalGroundedness, compliance pass rate, latencycompare_models.py
MemorySession EvalFollow-up accuracy, context retentionMulti-turn test suite

Critical: Always test with production-like data volume and concurrency. Latency and cost profiles change dramatically under load.

Common Pitfalls & Solutions

ProblemRoot CauseSolution
Wrong retrievalPoor chunking or missing metadataHybrid search + domain-tagged metadata
Compliance false rejectionsOverly strict validator promptCalibrate with false positive analysis
Infinite compliance loopsAnalyst ignores feedbackMax retry limit + escalation to human
Lost session contextMemory not persisted between requestsRedis/PostgreSQL-backed session store
High latencySequential agent callsParallel retrieval + streaming responses
Model works in eval but fails in prodDistribution shift in live queriesContinuous monitoring + golden set expansion

Future Enhancements

Conclusion: Building Trustworthy AI Through Rigorous Evaluation

Comparing LLMs for enterprise banking is not about finding the "smartest" model. It is about finding the model that delivers the highest rate of verified, compliant, auditable answers within operational constraints. This requires:

  1. A task-specific evaluation framework with golden test sets and domain-relevant metrics.

  2. A multi-agent architecture that separates concerns and enables granular measurement.

  3. Stateful orchestration via LangGraph that makes every reasoning step inspectable.

  4. Continuous validation through compliance agents and audit trails.

The ReguCheck system demonstrated here provides both the infrastructure for production deployment and the instrumentation needed for rigorous model comparison. In regulated industries, trust is not given—it is engineered, measured, and continuously validated.