Hybrid Credit Scoring: Where Classical Models End and LLMs Begin in Enterprise Fintech

Introduction

Credit scoring has historically been the domain of classical statistical models—logistic regression, gradient-boosted trees, and neural networks trained on tabular financial data. These models are exceptionally good at what they do: processing structured numerical features like payment history, debt-to-income ratios, and credit utilization to produce a deterministic risk score.

However, as fintech enterprises expanded into underserved markets and complex commercial lending, they hit a wall. Classical models cannot read a borrower's handwritten business plan, interpret the tone of an email dispute, or extract nuanced risk signals from a bank statement's transaction narratives. This is where Large Language Models (LLMs) entered the credit scoring stack—not to replace classical models, but to complement them.

This article explores how modern enterprise credit scoring systems divide responsibilities between classical ML and LLMs, and provides a complete Proof of Concept (POC) using a Multi-Agent LangGraph architecture with RAG and persistent state.

The Division of Labor: Classical Models vs. LLMs in Credit Scoring

When designing a hybrid credit scoring system, we made a clear architectural decision about which components each technology handles best:

Classical Models Handle:

• Structured Numerical Features: FICO scores, DTI ratios, credit utilization percentages, payment history counts.

• High-Frequency Transactional Data: Thousands of monthly transactions aggregated into behavioral features (e.g., "average daily balance," "overdraft frequency").

• Deterministic Risk Scoring: Producing the primary probability-of-default (PD) number with mathematical precision and regulatory auditability.

• Feature Engineering at Scale: Computing thousands of derived features from tabular data in milliseconds.

• Calibration and Thresholding: Mapping raw scores to risk tiers using isotonic regression or Platt scaling.

LLMs Handle:

• Unstructured Narrative Analysis: Reading business plans, owner's letters, and loan purpose statements to extract qualitative risk factors.

• Contextual Interpretation: Understanding that "revenue dropped 20%" means something different if the narrative explains it was due to a planned equipment upgrade versus a lost client.

• Policy Retrieval and Interpretation: Using RAG to fetch relevant underwriting policies and interpreting them against the specific application context.

• Explainability Generation: Translating complex model outputs into natural language explanations for loan officers and regulators.

• Ambiguity Resolution: Flagging contradictory information (e.g., application says "5 years in business" but bank statements show only 2 years of activity).

• Alternative Data Synthesis: Incorporating news articles, social media sentiment, or industry reports about the borrower's sector.

The Handoff Point:

The classical model produces a base risk score (e.g., "PD = 12%"). The LLM layer then produces a contextual adjustment vector (e.g., "+3% risk due to negative narrative sentiment" or "-2% risk due to strong business plan quality"). A final synthesis layer combines these into the decision score.

Real-Time Use Case: Small Business Loan Underwriting

The Scenario: A small business owner applies for a $75,000 working capital loan. The application includes:

• Structured data: 3 years of tax returns, business bank statements, personal credit report (FICO 680, DTI 38%).

• Unstructured data: A 2-page business plan narrative, a letter explaining a recent revenue dip, and 3 customer testimonials.

The Workflow:

1. The Classical ML Agent processes the structured data and outputs a base PD of 14% (moderate risk).

2. The RAG Policy Agent retrieves relevant underwriting policies for the borrower's industry (restaurant sector).

3. The LLM Narrative Agent reads the business plan and owner's letter, identifying that the revenue dip was due to a kitchen renovation (capital improvement, not demand loss) and extracting positive sentiment from customer testimonials.

4. The Synthesis Agent combines the classical score with the LLM's qualitative assessment, adjusts the PD down to 11%, and generates an explainable recommendation for the loan officer.

Enterprise Multi-Agent LangGraph Architecture

Our architecture uses LangGraph to orchestrate specialized agents, each handling its domain of expertise:

• Classical ML Agent: Simulates an XGBoost model scoring structured features.

• RAG Policy Agent: Retrieves industry-specific underwriting guidelines from a vector database.

• LLM Narrative Agent: Analyzes unstructured documents for qualitative risk signals.

• Synthesis Agent: Combines classical and LLM outputs into a final decision with explainability.

Persistent memory tracks the application state throughout the underwriting journey, enabling auditability and human-in-the-loop overrides.

Step-by-Step POC Implementation

Step 1: Defining State, Memory, and Hybrid Schema

We define the state to carry both classical model outputs and LLM-derived insights, with memory for audit trails.

# backend/graph_state.py
from typing import TypedDict, List, Annotated, Literal, Optional
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.memory import MemorySaver

class CreditScoringState(TypedDict):
    application_id: str
    structured_features: dict
    classical_pd: float  # Probability of Default from classical model
    business_plan_text: str
    owner_letter_text: str
    retrieved_policies: List[str]
    narrative_risk_adjustment: float  # LLM-derived adjustment (+ increases risk, - decreases)
    narrative_findings: List[str]
    final_pd: float
    decision: Literal["APPROVE", "REVIEW", "DECLINE"]
    explanation: str
    agent_trace: Annotated[List[str], "Audit trail of scoring process"]

memory = MemorySaver()

Step 2: Building the Multi-Agent Hybrid Workflow

We implement each agent, clearly showing where classical logic ends and LLM reasoning begins.

# backend/agents.py
from .graph_state import CreditScoringState, memory

# Mock Vector DB for underwriting policies
MOCK_POLICY_DB = {
    "restaurant": [
        "Policy REST-001: Restaurant sector loans require 2+ years operating history. Capital improvements (renovations) are viewed positively if documented.",
        "Policy REST-002: Customer testimonials and online reputation can offset up to 3% PD adjustment for borderline applications."
    ],
    "retail": [
        "Policy RET-001: Retail sector requires strong inventory turnover metrics."
    ]
}

def classical_ml_agent(state: CreditScoringState):
    """
    CLASSICAL MODEL: Processes structured numerical features.
    Simulates an XGBoost model trained on tabular credit data.
    """
    features = state["structured_features"]
    
    # Simulated XGBoost scoring logic (in production, this calls a trained model)
    base_pd = 0.05  # 5% base PD
    
    # Feature contributions (simplified logistic regression for demo)
    if features.get("fico_score", 700) < 650:
        base_pd += 0.04
    if features.get("dti_ratio", 0.3) > 0.4:
        base_pd += 0.03
    if features.get("years_in_business", 3) < 2:
        base_pd += 0.02
    if features.get("monthly_revenue", 50000) < 30000:
        base_pd += 0.02
    
    # Cap at reasonable range
    classical_pd = min(max(base_pd, 0.01), 0.95)
    
    state["agent_trace"].append(
        f"Classical ML Agent: Processed {len(features)} structured features. "
        f"Base PD = {classical_pd:.1%} (deterministic, auditable)"
    )
    
    return {"classical_pd": classical_pd}

def rag_policy_agent(state: CreditScoringState):
    """
    RAG: Retrieves relevant underwriting policies based on borrower's industry.
    """
    industry = state["structured_features"].get("industry", "restaurant")
    policies = MOCK_POLICY_DB.get(industry, ["No specific policies found."])
    
    state["agent_trace"].append(
        f"RAG Policy Agent: Retrieved {len(policies)} policies for {industry} sector."
    )
    
    return {"retrieved_policies": policies}

def llm_narrative_agent(state: CreditScoringState):
    """
    LLM: Analyzes unstructured narrative documents for qualitative risk signals.
    This is where LLMs excel - understanding context, sentiment, and nuance.
    """
    business_plan = state["business_plan_text"]
    owner_letter = state["owner_letter_text"]
    policies = state["retrieved_policies"]
    
    findings = []
    risk_adjustment = 0.0
    
    # Simulated LLM reasoning (in production, this calls an LLM with the documents)
    # The LLM would extract insights like:
    
    if "renovation" in owner_letter.lower() or "upgrade" in owner_letter.lower():
        findings.append("Owner letter indicates revenue dip due to capital improvement (renovation), not demand loss.")
        risk_adjustment -= 0.02  # Positive signal
    
    if "testimonial" in business_plan.lower() or "customer" in business_plan.lower():
        findings.append("Business plan includes strong customer testimonials and community support.")
        risk_adjustment -= 0.01  # Positive signal
    
    if "new location" in business_plan.lower() and "expansion" in business_plan.lower():
        findings.append("Business plan outlines expansion strategy with clear market analysis.")
        risk_adjustment -= 0.01  # Positive signal
    
    # Check for negative signals
    if "lawsuit" in business_plan.lower() or "dispute" in owner_letter.lower():
        findings.append("Narrative mentions legal disputes - potential risk factor.")
        risk_adjustment += 0.03  # Negative signal
    
    # Cap adjustment based on policy
    max_adjustment = 0.03  # Policy allows up to 3% adjustment
    risk_adjustment = max(min(risk_adjustment, max_adjustment), -max_adjustment)
    
    state["agent_trace"].append(
        f"LLM Narrative Agent: Analyzed unstructured documents. "
        f"Found {len(findings)} qualitative signals. Risk adjustment: {risk_adjustment:+.1%}"
    )
    
    return {
        "narrative_findings": findings,
        "narrative_risk_adjustment": risk_adjustment
    }

def synthesis_agent(state: CreditScoringState):
    """
    SYNTHESIS: Combines classical PD with LLM narrative adjustment.
    Generates final decision and explainable recommendation.
    """
    classical_pd = state["classical_pd"]
    narrative_adjustment = state["narrative_risk_adjustment"]
    findings = state["narrative_findings"]
    
    # Final PD = Classical PD + LLM Adjustment
    final_pd = classical_pd + narrative_adjustment
    final_pd = min(max(final_pd, 0.01), 0.95)
    
    # Decision logic
    if final_pd < 0.10:
        decision = "APPROVE"
    elif final_pd < 0.20:
        decision = "REVIEW"
    else:
        decision = "DECLINE"
    
    # Generate explainable recommendation
    explanation = f"Classical model PD: {classical_pd:.1%}. "
    if narrative_adjustment != 0:
        explanation += f"LLM narrative adjustment: {narrative_adjustment:+.1%}. "
    explanation += f"Final PD: {final_pd:.1%}. Decision: {decision}.\n\n"
    explanation += "Key qualitative findings:\n"
    for finding in findings:
        explanation += f"• {finding}\n"
    
    state["agent_trace"].append(
        f"Synthesis Agent: Combined classical ({classical_pd:.1%}) + LLM ({narrative_adjustment:+.1%}) = Final PD {final_pd:.1%}. Decision: {decision}"
    )
    
    return {
        "final_pd": final_pd,
        "decision": decision,
        "explanation": explanation
    }

# Build the Graph
workflow = StateGraph(CreditScoringState)

workflow.add_node("classical_ml", classical_ml_agent)
workflow.add_node("rag_policy", rag_policy_agent)
workflow.add_node("llm_narrative", llm_narrative_agent)
workflow.add_node("synthesis", synthesis_agent)

workflow.set_entry_point("classical_ml")
workflow.add_edge("classical_ml", "rag_policy")
workflow.add_edge("rag_policy", "llm_narrative")
workflow.add_edge("llm_narrative", "synthesis")
workflow.add_edge("synthesis", END)

app = workflow.compile(checkpointer=memory)

Step 3: The FastAPI Backend

We expose the hybrid scoring system via a REST API with persistent memory for audit trails.

# backend/main.py
from fastapi import FastAPI
from pydantic import BaseModel
from .agents import app

app_api = FastAPI(title="Hybrid Credit Scoring System")

class CreditApplication(BaseModel):
    application_id: str
    structured_features: dict
    business_plan_text: str
    owner_letter_text: str
    thread_id: str = "credit_audit_01"

@app_api.post("/score-application")
async def score_application(application: CreditApplication):
    config = {"configurable": {"thread_id": application.thread_id}}
    
    initial_state = {
        "application_id": application.application_id,
        "structured_features": application.structured_features,
        "classical_pd": 0.0,
        "business_plan_text": application.business_plan_text,
        "owner_letter_text": application.owner_letter_text,
        "retrieved_policies": [],
        "narrative_risk_adjustment": 0.0,
        "narrative_findings": [],
        "final_pd": 0.0,
        "decision": "",
        "explanation": "",
        "agent_trace": []
    }
    
    final_state = app.invoke(initial_state, config)
    
    return {
        "classical_pd": final_state["classical_pd"],
        "narrative_adjustment": final_state["narrative_risk_adjustment"],
        "final_pd": final_state["final_pd"],
        "decision": final_state["decision"],
        "explanation": final_state["explanation"],
        "agent_trace": final_state["agent_trace"]
    }

Step 4: The Streamlit Frontend

A dashboard for loan officers to submit applications and view the hybrid scoring breakdown.

# frontend/app.py
import streamlit as st
import requests

st.set_page_config(page_title="Hybrid Credit Scoring", layout="wide")
st.title("🏦 Hybrid Credit Scoring: Classical ML + LLM")
st.markdown("*Enterprise Multi-Agent LangGraph RAG System*")

st.sidebar.header("Application Details")
app_id = st.sidebar.text_input("Application ID", "APP-2026-001")
thread_id = st.sidebar.text_input("Audit Thread ID", "audit_001")

st.sidebar.subheader("Structured Features (Classical ML)")
fico = st.sidebar.slider("FICO Score", 300, 850, 680)
dti = st.sidebar.slider("DTI Ratio", 0.0, 1.0, 0.38, 0.01)
years_business = st.sidebar.slider("Years in Business", 0, 20, 3)
monthly_revenue = st.sidebar.number_input("Monthly Revenue ($)", 0, 500000, 45000)
industry = st.sidebar.selectbox("Industry", ["restaurant", "retail", "services"])

st.subheader("Unstructured Documents (LLM Analysis)")
business_plan = st.text_area(
    "Business Plan Excerpt",
    "Our restaurant has served the community for 3 years. We are planning an expansion to a new location downtown. Customer testimonials highlight our commitment to quality and local sourcing.",
    height=150
)

owner_letter = st.text_area(
    "Owner's Letter",
    "Revenue dipped last quarter due to a kitchen renovation and equipment upgrade. We expect full recovery by Q2 with increased capacity.",
    height=150
)

if st.button("Run Hybrid Credit Scoring"):
    with st.spinner("Classical ML and LLM agents are analyzing..."):
        structured_features = {
            "fico_score": fico,
            "dti_ratio": dti,
            "years_in_business": years_business,
            "monthly_revenue": monthly_revenue,
            "industry": industry
        }
        
        response = requests.post(
            "http://localhost:8000/score-application",
            json={
                "application_id": app_id,
                "structured_features": structured_features,
                "business_plan_text": business_plan,
                "owner_letter_text": owner_letter,
                "thread_id": thread_id
            }
        )
        
        if response.status_code == 200:
            data = response.json()
            
            col1, col2, col3 = st.columns(3)
            with col1:
                st.metric("Classical Model PD", f"{data['classical_pd']:.1%}")
                st.caption("XGBoost on structured features")
            with col2:
                st.metric("LLM Narrative Adjustment", f"{data['narrative_adjustment']:+.1%}")
                st.caption("Qualitative risk signals")
            with col3:
                st.metric("Final PD", f"{data['final_pd']:.1%}")
                if data["decision"] == "APPROVE":
                    st.success(f"  {data['decision']}")
                elif data["decision"] == "REVIEW":
                    st.warning(f"  {data['decision']}")
                else:
                    st.error(f"  {data['decision']}")
            
            st.subheader("Explainable Recommendation")
            st.info(data["explanation"])
            
            with st.expander("Agent Audit Trail"):
                for trace in data["agent_trace"]:
                    st.write(f"  {trace}")
        else:
            st.error("Failed to connect to scoring engine.")

Conclusion

Modern enterprise credit scoring systems are not choosing between classical models and LLMs they are strategically combining both. Classical models handle what they do best: processing structured numerical data at scale with mathematical precision and regulatory auditability. LLMs handle what they do best: understanding unstructured narratives, extracting contextual insights, and generating human-readable explanations.

In our hybrid architecture, the classical XGBoost model produces a deterministic probability of default from tabular features. The LLM layer then analyzes business plans, owner letters, and other unstructured documents to identify qualitative risk signals that classical models cannot see. Through RAG, the system retrieves relevant underwriting policies to ensure the LLM's adjustments align with institutional guidelines. The synthesis agent combines these outputs into a final decision with full explainability.

This division of labor ensures that credit scoring is both mathematically rigorous and contextually aware, enabling fintech enterprises to serve underserved markets while maintaining regulatory compliance and risk management standards.