AI Agents  

The Challenge of Contradictory Evidence in Financial RAG

Introduction

In enterprise financial systems, contradictory evidence is not an edge case; it is a daily reality. A naive RAG system that simply concatenates retrieved chunks and passes them to an LLM will either hallucinate a compromise, pick the first chunk it saw, or fail entirely when faced with conflicting data.

Why Contradictions Occur in Finance

  • Temporal Drift: A 2022 credit agreement states a 4.0x leverage limit, but a 2024 amendment changes it to 3.5x.

  • Hierarchical Conflicts: An internal risk policy sets a $50k transaction limit, but a signed client side-letter overrides it to $100k.

  • Regulatory Overlap: SEC guidance on a specific accounting treatment conflicts with internal GAAP interpretation.

  • Data Quality Issues: OCR errors in legacy PDFs or conflicting entries across different core banking systems.

The Resolution Framework

To handle this, we cannot rely on semantic similarity alone. We must implement a Deterministic + LLM Hybrid Resolution Framework:

  • Metadata-Driven Determinism: Use the metadata schema (from our previous architecture) to apply hard rules. Newer documents override older ones. Master agreements override side letters.

  • LLM-as-Judge (Semantic Resolution): If metadata doesn't resolve it (e.g., two documents from the same date), an LLM analyzes the semantic intent and flags the ambiguity.

  • Human-in-the-Loop (Escalation): If the contradiction involves high-risk financial thresholds and cannot be deterministically resolved, the system must pause and escalate to a human expert.

Real-Time Use Case

The Use Case: Automated Credit Underwriting & Covenant Reconciliation

Scenario: A loan officer is structuring a syndicated credit facility for "Acme Corp". The system must determine the client's Maximum Permitted Leverage Ratio (Debt/EBITDA) to price the loan correctly.

The Contradiction

The retrieval system pulls three different documents:

  • Internal Credit Memo (2023): States the max leverage is 4.0x.

  • External 10-K Filing (2024): Mentions a strict corporate policy capping leverage at 3.5x.

  • Credit Agreement Amendment (2024): A signed legal amendment temporarily bumps the limit to 4.5x to accommodate a specific acquisition.

The Goal

The multi-agent system must retrieve all three, detect the numerical and semantic contradiction, apply temporal and hierarchical resolution logic, and output a reconciled recommendation, flagging the temporary nature of the 4.5x limit for human review.

Enterprise Multi-Agent Architecture (LangGraph)

We use a Sequential Multi-Agent Pipeline with Conditional Escalation.

  • Retriever Agent: Fetches documents using our metadata-rich vector store.

  • Contradiction Detector Agent: Analyzes the retrieved chunks to identify conflicting numerical or semantic claims.

  • Conflict Resolver Agent: Applies business rules (temporal/hierarchical) to resolve the conflict. If unresolvable, it sets an escalation_required flag.

  • Underwriting Synthesizer Agent: Drafts the final credit memo based on the resolved facts.

State & Memory

  • State: UnderwritingState tracks the chunks, detected contradictions, resolution logic, and escalation flags.

  • Memory: LangGraph Checkpointer persists the state. If escalated to a human, the human's decision is injected back into the state to complete the graph.

mnm

Code Implementation

Prerequisites

pip install langgraph langchain langchain-openai pydantic

1. Define the State

from typing import Dict, List, Any, Literal, Optional
from typing_extensions import TypedDict
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.memory import MemorySaver
from langchain_core.messages import HumanMessage, SystemMessage
from langchain_openai import ChatOpenAI
import os

class UnderwritingState(TypedDict):
    client_id: str
    metric_query: str
    retrieved_chunks: List[Dict[str, Any]] # Contains text and metadata
    contradictions: List[Dict[str, str]]
    resolution_logic: str
    resolved_value: Optional[str]
    escalation_required: bool
    human_feedback: Optional[str]
    final_memo: str

llm = ChatOpenAI(model="gpt-4o", temperature=0)

2. Mock Enterprise Retrievers (Simulating the Contradiction)

def mock_retriever(query: str) -> List[Dict[str, Any]]:
    """Simulates retrieving three contradictory documents with rich metadata."""
    return [
        {
            "text": "The Borrower's Maximum Permitted Leverage Ratio shall not exceed 4.0x.",
            "metadata": {"doc_type": "internal_credit_memo", "date": "2023-05-10", "authority": "internal", "version": "1.0"}
        },
        {
            "text": "Acme Corp maintains a strict internal corporate policy capping consolidated leverage at 3.5x.",
            "metadata": {"doc_type": "10k_filing", "date": "2024-02-15", "authority": "external_regulatory", "version": "2.0"}
        },
        {
            "text": "Notwithstanding Section 4.1, the Maximum Permitted Leverage Ratio shall be increased to 4.5x for a period of 12 months to facilitate the Omega Acquisition.",
            "metadata": {"doc_type": "credit_agreement_amendment", "date": "2024-08-20", "authority": "legal_binding", "version": "2.1"}
        }
    ]

3. Define the Agent Nodes

def retriever_node(state: UnderwritingState) -> dict:
    """Fetches relevant chunks."""
    chunks = mock_retriever(state['metric_query'])
    return {"retrieved_chunks": chunks}

def contradiction_detector_node(state: UnderwritingState) -> dict:
    """Identifies conflicting claims in the retrieved chunks."""
    chunks_text = "\n".join([f"[{c['metadata']['doc_type']} ({c['metadata']['date']})]: {c['text']}" for c in state['retrieved_chunks']])
    
    prompt = f"""Analyze the following financial document excerpts regarding '{state['metric_query']}'.
    Identify any numerical or semantic contradictions.
    
    Excerpts:
    {chunks_text}
    
    Output a JSON list of contradictions. If none, return an empty list.
    Format: [{{"claim_1": "...", "claim_2": "...", "reason": "..."}}]
    """
    response = llm.invoke([SystemMessage(content=prompt)])
    
    # Simple parsing for demo (Use structured output in prod)
    contradictions = []
    if "4.0x" in response.content and "3.5x" in response.content:
        contradictions.append({"claim_1": "4.0x limit", "claim_2": "3.5x limit", "reason": "Conflicting leverage caps"})
    if "4.5x" in response.content:
        contradictions.append({"claim_1": "4.5x temporary limit", "claim_2": "Standard limits", "reason": "Amendment overrides standard limits"})
        
    return {"contradictions": contradictions}

def conflict_resolver_node(state: UnderwritingState) -> dict:
    """Applies deterministic and semantic logic to resolve contradictions."""
    if not state['contradictions']:
        return {"resolution_logic": "No contradictions found.", "escalation_required": False}
    
    # DETERMINISTIC LOGIC: Sort by date (newest first) and authority level
    sorted_chunks = sorted(
        state['retrieved_chunks'], 
        key=lambda x: (x['metadata']['date'], x['metadata']['authority']), 
        reverse=True
    )
    
    winning_chunk = sorted_chunks[0]
    
    # SEMANTIC LOGIC: Check if the winning chunk is a "temporary" or "conditional" override
    prompt = f"""Analyze the winning document excerpt:
    "{winning_chunk['text']}"
    
    Is this a permanent rule or a temporary/conditional exception? 
    Answer with "permanent" or "temporary" and explain why.
    """
    response = llm.invoke([SystemMessage(content=prompt)])
    
    is_temporary = "temporary" in response.content.lower()
    
    resolution = f"Resolved using temporal/hierarchical priority. Winning doc: {winning_chunk['metadata']['doc_type']} ({winning_chunk['metadata']['date']}). "
    resolution += f"LLM Analysis: {response.content}"
    
    # If it's a temporary override of a core risk metric, flag for human review
    escalation = is_temporary and "4.5x" in winning_chunk['text']
    
    return {
        "resolution_logic": resolution,
        "resolved_value": "4.5x (Temporary)",
        "escalation_required": escalation
    }

def underwriting_synthesizer_node(state: UnderwritingState) -> dict:
    """Drafts the final memo, incorporating human feedback if escalated."""
    human_note = state.get('human_feedback', "No human feedback provided.")
    
    prompt = f"""Draft a Credit Underwriting Memo for {state['client_id']}.
    Metric: {state['metric_query']}
    Resolved Value: {state['resolved_value']}
    Resolution Logic: {state['resolution_logic']}
    Human Reviewer Feedback: {human_note}
    
    Provide a professional summary of the leverage ratio, the contradiction found, how it was resolved, and the final approved limit.
    """
    response = llm.invoke([HumanMessage(content=prompt)])
    return {"final_memo": response.content}

4. Build and Compile the LangGraph with Human-in-the-Loop

def route_resolver(state: UnderwritingState) -> str:
    """Conditional edge: If escalation is required, pause for human. Otherwise, synthesize."""
    if state.get('escalation_required', False):
        return "human_review"
    return "synthesizer"

workflow = StateGraph(UnderwritingState)

# Add Nodes
workflow.add_node("retriever", retriever_node)
workflow.add_node("detector", contradiction_detector_node)
workflow.add_node("resolver", conflict_resolver_node)
workflow.add_node("synthesizer", underwriting_synthesizer_node)

# Set Entry Point
workflow.set_entry_point("retriever")

# Add Edges
workflow.add_edge("retriever", "detector")
workflow.add_edge("detector", "resolver")

# Conditional Edge from Resolver
workflow.add_conditional_edges(
    "resolver",
    route_resolver,
    {
        "human_review": "human_review", # This will trigger an interrupt
        "synthesizer": "synthesizer"
    }
)

workflow.add_edge("synthesizer", END)

# Compile with interrupt_before for Human-in-the-Loop
memory = MemorySaver()
graph = workflow.compile(
    checkpointer=memory,
    interrupt_before=["human_review"] # Pauses graph here for human input
)

5. Execute the Investigation (with Human Escalation)

if __name__ == "__main__":
    initial_state = {
        "client_id": "ACME-CORP-99",
        "metric_query": "Maximum Permitted Leverage Ratio",
        "retrieved_chunks": [],
        "contradictions": [],
        "resolution_logic": "",
        "resolved_value": None,
        "escalation_required": False,
        "human_feedback": None,
        "final_memo": ""
    }
    
    config = {"configurable": {"thread_id": "ACME-UNDERWRITING-001"}}
    
    print("--- Starting Credit Underwriting Graph ---\n")
    
    # Run the graph until it hits the interrupt
    for event in graph.stream(initial_state, config):
        for node_name, node_output in event.items():
            print(f"[Node Executed: {node_name}]")
            
    # Check if the graph is paused for human review
    state_snapshot = graph.get_state(config)
    if state_snapshot.next and "human_review" in state_snapshot.next:
        print("\n[SYSTEM ALERT] Contradiction requires human review!")
        print(f"Resolved Value: {state_snapshot.values['resolved_value']}")
        print(f"Logic: {state_snapshot.values['resolution_logic']}")
        
        # Simulate Human Underwriter providing feedback
        human_decision = "Approved. Note that the 4.5x limit expires on 2025-08-20 and reverts to 3.5x."
        
        print("\n[Human Input Received] Resuming graph...")
        
        # Resume the graph with human feedback
        for event in graph.stream(
            {"human_feedback": human_decision}, 
            config,
            stream_mode="values" # Stream the final state
        ):
            if "final_memo" in event:
                print("\n" + "="*60)
                print("FINAL UNDERWRITING MEMO:")
                print("="*60)
                print(event['final_memo'])
                print("="*60)

Enterprise Production Considerations

To deploy this contradiction-handling framework in a Tier-1 financial institution, implement the following:

  • Structured Output for Contradiction Detection: In the contradiction_detector_node, use LangChain's with_structured_output with a Pydantic model. This guarantees the LLM returns a parseable JSON array of contradictions, preventing graph crashes due to malformed text.

  • Deterministic Rule Engine (Drools/Custom): While the LLM is great for semantic analysis, financial resolution rules (e.g., "Amendments always override original agreements") should be codified in a deterministic rule engine before passing the context to the LLM. The LLM should only handle edge cases where metadata is ambiguous.

  • Audit Trail for Resolution Logic: The resolution_logic string in the state is critical. In production, log this to an immutable audit database. Regulators (e.g., OCC, FCA) will ask why the system chose the 4.5x limit over the 3.5x limit. The exact LLM reasoning and metadata sorting logic must be preserved.

  • Feedback Loop for Vector Store: If a human overrides the system's resolution (e.g., "Actually, the 3.5x limit applies because the acquisition was cancelled"), this feedback must be fed back into the vector store. Update the metadata of the 2024 Amendment to include "status": "voided_by_human" so future retrievals don't make the same mistake.

  • Multi-Tenancy & Data Isolation: Ensure the retriever_node injects a tenant_id or client_id into the metadata filter. Contradictions must only be evaluated within the context of the specific client's document corpus to prevent cross-contamination.

Summary

Enterprise financial AI systems frequently encounter contradictory information from multiple authoritative sources. By combining deterministic metadata-driven rules, LLM-based semantic reasoning, and human-in-the-loop escalation within a LangGraph multi-agent workflow, organizations can resolve conflicts in a transparent and auditable manner. This hybrid approach improves the reliability of financial decision-making while satisfying regulatory, governance, and operational requirements for production-grade AI systems.p[;'\/¥p[;'\/p[;'\/963.*-p[;'\/p[;'\/

Membership not found