Prompt Design Lessons from Production Agent Failures: A Fintech Fixed Deposit Case Study

The Hard Truth About Enterprise Agent Prompts

After deploying multi-agent RAG systems across three fintech production environments, I learned that agent failures are rarely model failures—they are prompt architecture failures. The LLM is a compliant executor; when it hallucinates FD rates, misroutes compliance checks, or loses conversation state, the root cause is almost always how we structured cognitive instructions.

This article distills seven hard-won prompt design principles from production incidents in Fixed Deposit (FD) modules, implemented as a complete LangGraph multi-agent system with memory, state management, and audit-grade observability.

Seven Prompt Design Lessons from FD Production Incidents

Lesson 1: Role Prompts Must Include Negative Constraints

Failure: Advisor agent recommended a 5-year tax-saver FD to an NRI customer. NRIs are ineligible for Section 80C tax-saver FDs under FEMA regulations. The agent knew the rule but didn't apply it because the prompt said "recommend suitable FDs" without explicit exclusions.

Principle: Every role prompt must contain a NEVER section with domain-specific prohibitions, not just positive instructions.

Lesson 2: Retrieval Prompts Must Specify Document Schema, Not Just Intent

Failure: Retriever returned generic "FD interest rate" pages instead of the current quarter's rate card because the query was semantically similar but structurally wrong. Rate cards have a specific schema: {tenure_months, min_amount, senior_citizen_rate, effective_date}.

Principle: Retrieval prompts must reference the exact document structure expected downstream, not just natural language intent.

Lesson 3: Reasoning Prompts Must Force Structured Output Before Narrative

Failure: Compliance checker produced verbose reasoning that buried the actual pass/fail verdict in paragraph three. Downstream nodes couldn't reliably parse the decision.

Principle: Always require JSON/schema output first, then optional narrative explanation. Parse the structure; log the narrative.

Lesson 4: State Summarization Prompts Must Preserve Regulatory Facts Verbatim

Failure: Memory compressor paraphrased "RBI master direction RBI/2023-24/156" as "recent RBI guidelines." When the auditor agent later cited this, it was legally meaningless.

Principle: Compression prompts must distinguish between compressible context and immutable regulatory references. The latter are copied verbatim.

Lesson 5: Tool Selection Prompts Must Include Failure Mode Descriptions

Failure: Calculator agent called get_fd_rate with a tenure of 37 months. The API only accepts standard tenures (12, 15, 18, 24, 36, 48, 60). The error message confused the reasoner into retrying with 38 months.

Principle: Tool descriptions must include valid parameter ranges AND what happens when they're violated. Teach the agent the API's contract, not just its purpose.

Lesson 6: Multi-Agent Handoff Prompts Must Carry Explicit Context Contracts

Failure: Eligibility agent passed "customer_age": 62 to the advisor agent. Advisor treated this as exact age rather than "age at FD maturity," recommending a product with upper age limit 60. The handoff lacked semantic metadata.

Principle: Inter-agent messages must include field-level semantics via structured schemas, not bare key-value pairs.

Lesson 7: Guardrail Prompts Must Be Evaluated Separately from Task Prompts

Failure: We embedded RBI compliance checks inside the advisor prompt. When we updated the advisor's recommendation logic, we accidentally weakened the compliance check. Coupled prompts create regression risk.

Principle: Guardrails are independent evaluation nodes with frozen prompts. Task prompts evolve; guardrail prompts are version-controlled regulatory artifacts.

Real-Time Use Case: Fixed Deposit Advisory & Booking Pipeline

Scenario: A retail banking customer asks: "I'm 58, retiring next year. I have ₹15 lakhs maturing from an existing FD. What should I do for regular post-retirement income while staying tax-efficient?"

This triggers a regulated workflow:

Eligibility Check → KYC status, age, residency, existing FD portfolio
Rate & Product Retrieval → Current rate card, senior citizen premiums, tax-saver eligibility
Advisory Reasoning → Income projection, tax impact, laddering strategy
Compliance Validation → RBI suitability rules, disclosure requirements
Booking Preparation → Pre-fill application, generate term sheet

Each step applies the seven lessons above.

Step 1: State Schema with Semantic Field Contracts (Lesson 6)

# state.py
from typing import Annotated, Literal, Optional
from typing_extensions import TypedDict
from langgraph.graph.message import add_messages
from langchain_core.documents import Document
from pydantic import BaseModel, Field

class CustomerProfile(BaseModel):
    """Semantic contract for customer data across agents."""
    customer_id: str
    age_at_evaluation: int = Field(description="Current age in years")
    age_at_maturity: Optional[int] = Field(
        default=None, 
        description="Projected age when proposed FD matures. Critical for senior citizen eligibility."
    )
    residency_status: Literal["resident", "nri", "oci"]
    kyc_verified: bool
    pan_number: str
    existing_fd_portfolio: list[dict] = Field(default_factory=list)
    annual_taxable_income: float = Field(description="INR, current FY estimate")

class FDRecommendation(BaseModel):
    """Structured output from advisor - parsed before narrative."""
    product_code: str
    tenure_months: int
    principal_amount: float
    applicable_rate: float = Field(description="% p.a., including senior citizen premium if applicable")
    monthly_income_projected: float
    tax_treatment: Literal["taxable", "80c_deduction", "exempt"]
    laddering_strategy: Optional[list[dict]] = None
    regulatory_references: list[str] = Field(
        description="Verbatim RBI/master direction citations. NEVER paraphrase."
    )

class ComplianceVerdict(BaseModel):
    """Guardrail output - separate from advisory reasoning."""
    is_suitable: bool
    violations: list[str] = Field(default_factory=list)
    required_disclosures: list[str] = Field(default_factory=list)
    regulation_version: str = Field(description="Exact RBI circular number and date")

class FDAdvisoryState(TypedDict):
    messages: Annotated[list, add_messages]
    
    # Customer context (immutable after eligibility check)
    customer_profile: Optional[CustomerProfile]
    
    # Retrieval artifacts
    retrieved_rate_cards: list[Document]
    retrieved_product_specs: list[Document]
    retrieval_query_metadata: dict  # Tracks what was searched and why
    
    # Advisory artifacts
    recommendations: list[FDRecommendation]
    advisory_narrative: str
    
    # Compliance artifacts (separate concern - Lesson 7)
    compliance_verdict: Optional[ComplianceVerdict]
    
    # Execution artifacts
    booking_reference: Optional[str]
    term_sheet_url: Optional[str]
    
    # Memory & audit
    thread_id: str
    advisor_id: str
    regulatory_snapshot_date: str  # Date of rate/regulation capture

Step 2: Version-Controlled Prompt Templates

prompts/eligibility.md — Negative Constraints (Lesson 1)

# FD Eligibility Checker

## ROLE
Determine customer eligibility for Fixed Deposit products per RBI Master Direction 
RBI/2023-24/156 and bank internal policy FD-POL-2026-03.

## REQUIRED OUTPUT (JSON ONLY - Lesson 3)
{
  "eligible_products": ["product_code_1", ...],
  "ineligible_products": [{"code": "...", "reason": "..."}],
  "senior_citizen_eligible": boolean,
  "nri_restrictions": ["restriction_1", ...] | null,
  "regulatory_references": ["verbatim citation 1", ...]
}

## NEVER (Negative Constraints)
- NEVER recommend Tax Saver FD (80C) to NRI or OCI customers
- NEVER assume senior citizen rate applies before age 60 AT MATURITY (not current age)
- NEVER proceed if KYC is not verified - return eligible_products: []
- NEVER use outdated rate cards - verify effective_date matches regulatory_snapshot_date
- NEVER infer residency status from name or address patterns

prompts/retriever.md — Document Schema Awareness (Lesson 2)

# FD Document Retriever

## TASK
Retrieve documents matching the structured query below. Return ONLY documents 
that match the EXPECTED SCHEMA. Discard all others.

## EXPECTED DOCUMENT SCHEMAS
Rate Card: {bank_code, effective_date, tenure_months, min_amount, 
            standard_rate, senior_citizen_premium_bps, nri_rate_override}
Product Spec: {product_code, product_name, min_tenure, max_tenure, 
               premature_penalty_formula, tax_treatment, eligibility_criteria}
Regulation: {circular_number, issue_date, section, verbatim_text}

## QUERY
{{retrieval_query}}

## FILTERING RULES
- Rate cards: ONLY where effective_date <= {{regulatory_snapshot_date}}
- Product specs: ONLY where product_code IN {{eligible_products}}
- Regulations: ONLY exact circular matches, never summaries

prompts/advisor.md — Structured Output First (Lesson 3)

# FD Advisory Reasoner

## INPUT
Customer Profile: {{customer_profile_json}}
Retrieved Rate Cards: {{rate_cards_summary}}
Retrieved Product Specs: {{product_specs_summary}}
Eligible Products: {{eligible_products}}

## REQUIRED OUTPUT FORMAT
You MUST output valid JSON matching FDRecommendation schema FIRST.
After the JSON block, you may provide a customer-facing narrative.

### JSON BLOCK (parsed programmatically)
```json
{
  "product_code": "...",
  "tenure_months": ...,
  ...all FDRecommendation fields...
}

NARRATIVE (logged, not parsed)

Explain the recommendation in plain language. Reference specific rate card
entries and regulatory clauses by their EXACT identifiers.

REASONING REQUIREMENTS

Calculate projected monthly income using: (principal × rate × tenure) / (tenure × 12)
If customer retires within 12 months, evaluate laddering across 12/24/36 month tenures
Apply senior citizen premium ONLY if age_at_maturity >= 60
Cite rate card effective_date in every rate reference


#### `prompts/guardrails.md` — Independent Compliance (Lesson 7)
```markdown
# FD Suitability Guardrail

## PURPOSE
Independent compliance validation. This prompt is VERSION-CONTROLLED 
and modified ONLY by the compliance team. Advisory prompts must NOT 
contain compliance logic.

## INPUT
Recommendation: {{recommendation_json}}
Customer Profile: {{customer_profile_json}}
Regulatory Snapshot: {{regulatory_snapshot_date}}

## CHECKLIST (evaluate ALL)
1. Tenure within product spec min/max range
2. Amount meets minimum threshold for quoted rate tier
3. Senior citizen rate applied correctly based on age_at_maturity
4. NRI restrictions respected per FEMA Schedule 5
5. Tax treatment matches product specification exactly
6. All required disclosures present for customer segment
7. Recommendation dated within rate card validity period

## OUTPUT (JSON ONLY)
ComplianceVerdict schema. No narrative. No recommendations. 
Only pass/fail with specific violation codes.

Step 3: Cognitive Nodes Implementing the Lessons

# nodes/eligibility_node.py
import json
from pathlib import Path
from langchain_openai import ChatOpenAI
from fd_advisory.state import FDAdvisoryState, CustomerProfile

ELIGIBILITY_PROMPT = Path("prompts/eligibility.md").read_text()
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

async def eligibility_node(state: FDAdvisoryState) -> dict:
    # Extract profile from conversation history or CRM tool
    profile = await extract_customer_profile(state["messages"])
    
    response = await llm.ainvoke([
        {"role": "system", "content": ELIGIBILITY_PROMPT},
        {"role": "user", "content": json.dumps(profile.model_dump())}
    ])
    
    result = json.loads(response.content)
    
    # LESSON 4: Preserve regulatory references verbatim in state
    return {
        "customer_profile": profile,
        "messages": [{
            "role": "assistant", 
            "content": f"Eligibility checked. {len(result['eligible_products'])} products available.",
            "additional_kwargs": {"eligibility_result": result}
        }]
    }

# nodes/retriever_node.py
RETRIEVER_PROMPT = Path("prompts/retriever.md").read_text()

async def retriever_node(state: FDAdvisoryState) -> dict:
    eligibility = state["messages"][-1].additional_kwargs["eligibility_result"]
    
    # LESSON 2: Query includes expected schema, not just intent
    structured_query = {
        "retrieval_query": f"Rate cards and product specs for: {eligibility['eligible_products']}",
        "eligible_products": eligibility["eligible_products"],
        "regulatory_snapshot_date": state["regulatory_snapshot_date"]
    }
    
    rendered_prompt = render_template(RETRIEVER_PROMPT, structured_query)
    
    # Scoped retrieval with metadata filtering
    rate_docs = await rate_card_retriever.asimilarity_search(
        structured_query["retrieval_query"],
        k=3,
        filter={
            "doc_type": "rate_card",
            "effective_date_lte": state["regulatory_snapshot_date"]
        }
    )
    product_docs = await product_spec_retriever.asimilarity_search(
        structured_query["retrieval_query"],
        k=5,
        filter={"product_code": {"$in": eligibility["eligible_products"]}}
    )
    
    return {
        "retrieved_rate_cards": rate_docs,
        "retrieved_product_specs": product_docs,
        "retrieval_query_metadata": structured_query
    }

# nodes/compliance_guardrail_node.py (LESSON 7: Separate, frozen prompt)
GUARDRAIL_PROMPT = Path("prompts/guardrails.md").read_text()
guardrail_llm = ChatOpenAI(model="gpt-4o", temperature=0)  # Stronger model for compliance

async def compliance_guardrail_node(state: FDAdvisoryState) -> dict:
    # Guardrail receives STRUCTURED recommendation, not narrative
    rec_json = json.dumps(state["recommendations"][0].model_dump())
    profile_json = json.dumps(state["customer_profile"].model_dump())
    
    response = await guardrail_llm.ainvoke([
        {"role": "system", "content": GUARDRAIL_PROMPT},
        {"role": "user", "content": json.dumps({
            "recommendation_json": rec_json,
            "customer_profile_json": profile_json,
            "regulatory_snapshot_date": state["regulatory_snapshot_date"]
        })}
    ])
    
    verdict = ComplianceVerdict.model_validate_json(response.content)
    
    # LESSON 4: Regulation version preserved verbatim
    return {"compliance_verdict": verdict}

Step 4: Graph Assembly with Conditional Routing & Memory

# graph.py
from langgraph.graph import StateGraph, END
from langgraph.checkpoint.postgres.aio import AsyncPostgresSaver
from fd_advisory.state import FDAdvisoryState

def build_fd_advisory_graph():
    g = StateGraph(FDAdvisoryState)
    
    # Add cognitive nodes
    g.add_node("eligibility", eligibility_node)
    g.add_node("retriever", retriever_node)
    g.add_node("advisor", advisor_node)
    g.add_node("compliance_guardrail", compliance_guardrail_node)
    g.add_node("booking_prep", booking_prep_node)
    
    # Entry
    g.set_entry_point("eligibility")
    
    # Eligibility → Retriever (only if eligible products exist)
    g.add_conditional_edges("eligibility", 
        lambda s: "retriever" if has_eligible_products(s) else "END",
        {"retriever": "retriever", "END": END}
    )
    
    # Retriever → Advisor
    g.add_edge("retriever", "advisor")
    
    # Advisor → Compliance Guardrail (ALWAYS, never skip)
    g.add_edge("advisor", "compliance_guardrail")
    
    # Compliance → Booking or Back to Advisor
    g.add_conditional_edges("compliance_guardrail",
        lambda s: "booking_prep" if s["compliance_verdict"].is_suitable else "advisor",
        {"booking_prep": "booking_prep", "advisor": "advisor"}
    )
    
    # Booking → End
    g.add_edge("booking_prep", END)
    
    # Enterprise memory with Postgres checkpoints
    checkpointer = AsyncPostgresSaver.from_conn_string(
        "postgresql://fd_advisory:***@pg-cluster:5432/langgraph_checkpoints"
    )
    
    return g.compile(checkpointer=checkpointer)

Step 5: Production Invocation with Audit Trail

# main.py
import asyncio
from datetime import date

async def handle_fd_advisory_request(customer_message: str, thread_id: str, advisor_id: str):
    app = build_fd_advisory_graph()
    
    config = {
        "configurable": {
            "thread_id": thread_id,
            "checkpoint_ns": "fd_advisory_v2"  # Version namespace for prompt upgrades
        }
    }
    
    result = await app.ainvoke({
        "messages": [{"role": "user", "content": customer_message}],
        "thread_id": thread_id,
        "advisor_id": advisor_id,
        "regulatory_snapshot_date": date.today().isoformat()
    }, config=config)
    
    # Audit: compliance verdict is always present and structured
    if result.get("compliance_verdict"):
        await log_compliance_audit(
            thread_id=thread_id,
            verdict=result["compliance_verdict"],
            regulatory_snapshot=result["regulatory_snapshot_date"]
        )
    
    return result

# Usage
result = asyncio.run(handle_fd_advisory_request(
    customer_message="I'm 58, retiring next year. ₹15L maturing. Need post-retirement income, tax-efficient.",
    thread_id="cust-2026-08-09-FD-4471",
    advisor_id="advisor_priya_sharma"
))

Observability: Mapping Lessons to Metrics

Lesson	Production Metric	Alert Threshold
1. Negative constraints	Ineligible product recommendation rate	> 0% (zero tolerance)
2. Schema-aware retrieval	Retrieval schema mismatch rate	> 5%
3. Structured output first	JSON parse failure rate in advisor output	> 2%
4. Verbatim regulatory refs	Paraphrased citation detected in audit	> 0%
5. Tool failure modes	Invalid parameter retries per session	> 3
6. Semantic handoff contracts	Cross-agent field misinterpretation flags	> 1%
7. Separate guardrails	Compliance verdict changed after advisory prompt update	Track as regression signal

Key Takeaways for Fintech Agent Teams

Prompts are regulatory artifacts. Treat them like policy documents: version-controlled, reviewed by compliance, change-managed. Your guardrails.md is as important as your RBI circular library.
Structure beats eloquence. Every inter-node communication should be parseable without regex. If you're writing string parsing code to extract agent outputs, your prompt design has failed.
Negative knowledge is more valuable than positive knowledge. Telling an FD advisor what it cannot recommend prevents regulatory breaches more reliably than telling it what it should recommend.
Memory compression must respect legal semantics. You can summarize customer preferences; you cannot summarize RBI circular numbers. Build this distinction into your summarization prompts explicitly.
Test prompts against failure modes, not happy paths. Your eval suite should include NRI customers asking for 80C FDs, 59-year-olds expecting senior rates, and requests referencing superseded circulars. These are the cases that reveal prompt architecture weaknesses.

The separation of planning, retrieval, reasoning, and execution isn't just good engineering—it's a regulatory requirement in financial services. When each cognitive function has its own prompt, its own evaluation criteria, and its own audit trail, you transform agent behavior from emergent to engineered. That's the difference between a demo and a production fintech system.