Introduction

In the realm of Generative AI, it is dangerously easy to be impressed by eloquence. Large Language Models (LLMs) are trained on vast corpora of text, making them exceptionally good at sounding professional, confident, and coherent. However, in high-stakes domains like Tax Management, linguistic fluency is not just insufficient it can be deceptive. A model can generate a perfectly structured, grammatically flawless response that cites non-existent tax codes or misinterprets critical compliance deadlines. This article addresses a critical challenge for AI engineers: How do we evaluate LLM responses for domain correctness, not just linguistic fluency? We will move beyond simple "does it sound right?" checks and implement a robust evaluation strategy within an enterprise-grade architecture.

We will build a complete Proof of Concept (POC) for a Tax Management Intelligence Copilot. This system uses LangGraph to orchestrate multiple specialized agents, Retrieval-Augmented Generation (RAG) to ground answers in verified tax documents, and persistent Memory/State to handle complex, multi-turn queries. By the end of this guide, you will have a working full-stack application that demonstrates how to architect for accuracy, traceability, and domain-specific validation.

Disclaimer: This POC uses synthetic tax data for educational purposes. It is not a substitute for professional tax advice.

The Trap of Linguistic Fluency in Tax Management

Imagine a tax analyst asks: "What is the TDS rate for technical services paid to a US entity under the India-US DTAA?"

An LLM might respond: "According to the latest regulations, the TDS rate is 10% as per Article 12."

This sounds authoritative. It uses correct terminology (TDS, DTAA, Article 12). But is it correct?

If the model is merely "fluent," it might guess based on pattern matching. If it is "domain-correct," it must retrieve the specific treaty text, apply the logic, and cite the source. Our evaluation framework must catch the difference.

Defining Domain Correctness vs. Linguistic Fluency

Feature

Linguistic Fluency

Domain Correctness

Focus

Grammar, Tone, Structure

Factual Accuracy, Logic, Compliance

Failure Mode

Awkward phrasing, typos

Hallucination, Misinterpretation, Outdated Info

Evaluation

Perplexity, Human Readability

Groundedness, Citation Accuracy, Logic Verification

Fix

Prompt Engineering

RAG, Knowledge Graph, Specialized Agents

Evaluation Framework for Tax AI

To evaluate domain correctness, we implement a multi-layered check in our POC:

  1. Groundedness Check: Does every claim in the answer map back to a retrieved document chunk?

  2. Citation Completeness: Are specific sections/clauses cited?

  3. Negative Constraint Adherence: Did the model avoid inventing rules when information was missing?

  4. Logical Consistency: Do the calculated implications follow the retrieved rules?

We achieve this not by asking the user to judge, but by building a Compliance Agent into our LangGraph workflow that acts as an internal auditor before the response reaches the user.

Real-Time Use Case: Cross-Border Transaction Compliance

Scenario: A multinational corporation (MNC) is planning to repatriate dividends from its Indian subsidiary to its parent company in Singapore. The tax team needs to know:

  1. The applicable Withholding Tax (WHT) rate.

  2. Required forms (e.g., Form 15CA/CB).

  3. Any specific conditions under the India-Singapore DTAA.

Query: "We are repatriating dividends from India to Singapore. What is the WHT rate and what documentation is needed?"

System Behavior:

Technology Stack Overview

Project Architecture and Folder Structure

tax-ai-evaluator/
├── backend/
│   ├── app/
│   │   ├── main.py
│   │   ├── config.py
│   │   ├── models/
│   │   │   └── state.py
│   │   ├── agents/
│   │   │   ├── retriever.py
│   │   │   ├── analyst.py
│   │   │   └── compliance.py
│   │   ├── graph/
│   │   │   └── workflow.py
│   │   └── rag/
│   │       └── ingest.py
│   └── requirements.txt
├── frontend/
│   ├── src/
│   │   ├── App.jsx
│   │   └── components/
│   │       ├── Chat.jsx
│   │       └── EvidencePanel.jsx
│   └── package.json
└── README.md

Step-by-Step POC Implementation

1. Backend Setup and Configuration

backend/requirements.txt

fastapi
uvicorn
langgraph
langchain
langchain-openai
langchain-community
chromadb
sentence-transformers
python-dotenv
pydantic

backend/app/config.py

import os
from dotenv import load_dotenv

load_dotenv()

class Settings:
    OPENAI_API_KEY = os.getenv("OPENAI_API_KEY")
    MODEL_NAME = "gpt-4o-mini"
    CHROMA_PATH = "./chroma_db"

settings = Settings()

2. Building the Tax Knowledge Base (RAG)

We create synthetic but realistic tax documents to simulate a real knowledge base.

backend/app/rag/ingest.py

from langchain_core.documents import Document
from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings
from langchain_text_splitters import RecursiveCharacterTextSplitter

def get_synthetic_tax_docs():
    return [
        Document(
            page_content="Under Article 10 of the India-Singapore DTAA, dividends paid by an Indian company to a Singapore resident are taxed at 10% if the beneficial owner holds at least 25% of the capital. Otherwise, the rate is 15%. This is subject to the Limitation of Benefits (LOB) clause.",
            metadata={"source": "India-Singapore DTAA", "article": "10", "jurisdiction": "India-Singapore"}
        ),
        Document(
            page_content="Section 115A of the Income Tax Act specifies that dividends received by foreign companies are taxable at 20% plus surcharge and cess, unless a lower rate is provided under a DTAA.",
            metadata={"source": "Income Tax Act 1961", "section": "115A", "jurisdiction": "India"}
        ),
        Document(
            page_content="For remittance of dividends, Form 15CA and Form 15CB are mandatory. Form 15CB requires certification by a Chartered Accountant confirming the correct tax deduction.",
            metadata={"source": "RBI Guidelines", "form": "15CA/15CB", "jurisdiction": "India"}
        )
    ]

def ingest_data():
    docs = get_synthetic_tax_docs()
    splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
    split_docs = splitter.split_documents(docs)
    
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    vector_store = Chroma.from_documents(
        documents=split_docs,
        embedding=embeddings,
        persist_directory="./chroma_db"
    )
    print("Tax knowledge base ingested.")
    return vector_store

3. Defining LangGraph State and Memory

State management is crucial for tracking the flow of information and maintaining conversation history.

backend/app/models/state.py

from typing import TypedDict, List, Optional, Annotated
from langgraph.graph import add_messages

class TaxAgentState(TypedDict):
    messages: Annotated[list, add_messages]
    user_query: str
    retrieved_context: List[dict]
    draft_answer: str
    compliance_check_passed: bool
    compliance_feedback: str
    final_response: str
    citations: List[dict]
    needs_human_review: bool

4. Implementing Multi-Agent Nodes

Retriever Agent

backend/app/agents/retriever.py

from langchain_community.vectorstores import Chroma
from langchain_community.embeddings import HuggingFaceEmbeddings

def retrieval_node(state):
    embeddings = HuggingFaceEmbeddings(model_name="all-MiniLM-L6-v2")
    db = Chroma(persist_directory="./chroma_db", embedding_function=embeddings)
    
    docs = db.similarity_search(state["user_query"], k=3)
    state["retrieved_context"] = [
        {"content": d.page_content, "metadata": d.metadata} for d in docs
    ]
    return state

Analyst Agent

backend/app/agents/analyst.py

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from app.config import settings

llm = ChatOpenAI(model=settings.MODEL_NAME, temperature=0)

prompt = ChatPromptTemplate.from_template("""
You are a Senior Tax Consultant. Answer the query using ONLY the provided context.
If the context does not contain the answer, state that information is missing.
Cite specific articles or sections.

Context:
{context}

Query: {query}

Answer:
""")

def analyst_node(state):
    context_text = "\n".join([doc["content"] for doc in state["retrieved_context"]])
    chain = prompt | llm
    response = chain.invoke({"context": context_text, "query": state["user_query"]})
    state["draft_answer"] = response.content
    return state

Compliance Agent (The Evaluator)

This is where we evaluate domain correctness. backend/app/agents/compliance.py

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from app.config import settings

llm = ChatOpenAI(model=settings.MODEL_NAME, temperature=0)

compliance_prompt = ChatPromptTemplate.from_template("""
You are a Tax Compliance Auditor. Review the draft answer against the provided context.
Check for:
1. Hallucinations: Claims not supported by context.
2. Missing Citations: Failure to cite specific articles/sections.
3. Critical Omissions: Missing key conditions (e.g., LOB clause, holding %).

If the answer is compliant, return "PASS".
If not, return "FAIL" and explain why.

Context:
{context}

Draft Answer:
{answer}

Verdict:
""")

def compliance_node(state):
    context_text = "\n".join([doc["content"] for doc in state["retrieved_context"]])
    chain = compliance_prompt | llm
    result = chain.invoke({"context": context_text, "answer": state["draft_answer"]})
    
    verdict = result.content.strip()
    if "PASS" in verdict:
        state["compliance_check_passed"] = True
        state["final_response"] = state["draft_answer"]
        state["needs_human_review"] = False
    else:
        state["compliance_check_passed"] = False
        state["compliance_feedback"] = verdict
        state["final_response"] = "⚠️ Compliance Check Failed: " + verdict
        state["needs_human_review"] = True
        
    # Extract citations for UI
    state["citations"] = state["retrieved_context"]
    return state

5. Orchestrating the Workflow

backend/app/graph/workflow.py

from langgraph.graph import StateGraph, END
from app.models.state import TaxAgentState
from app.agents.retriever import retrieval_node
from app.agents.analyst import analyst_node
from app.agents.compliance import compliance_node
from langgraph.checkpoint.sqlite import SqliteSaver

def create_workflow():
    workflow = StateGraph(TaxAgentState)
    
    workflow.add_node("retrieve", retrieval_node)
    workflow.add_node("analyze", analyst_node)
    workflow.add_node("audit", compliance_node)
    
    workflow.set_entry_point("retrieve")
    workflow.add_edge("retrieve", "analyze")
    workflow.add_edge("analyze", "audit")
    workflow.add_edge("audit", END)
    
    # Persistent Memory
    memory = SqliteSaver.from_conn_string(":memory:")
    app = workflow.compile(checkpointer=memory)
    return app

tax_graph = create_workflow()

6. FastAPI Endpoints

backend/app/main.py

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel
from app.graph.workflow import tax_graph
from app.rag.ingest import ingest_data
import uuid

app = FastAPI(title="Tax AI Evaluator")

class QueryRequest(BaseModel):
    question: str
    conversation_id: str = None

@app.post("/ingest")
def ingest():
    ingest_data()
    return {"status": "Knowledge Base Updated"}

@app.post("/chat")
def chat(request: QueryRequest):
    thread_id = request.conversation_id or str(uuid.uuid4())
    config = {"configurable": {"thread_id": thread_id}}
    
    try:
        result = tax_graph.invoke(
            {"messages": [{"role": "user", "content": request.question}], "user_query": request.question},
            config=config
        )
        return {
            "conversation_id": thread_id,
            "response": result["final_response"],
            "compliance_passed": result["compliance_check_passed"],
            "citations": result["citations"],
            "needs_review": result["needs_human_review"]
        }
    except Exception as e:
        raise HTTPException(status_code=500, detail=str(e))

Frontend Implementation with React

frontend/src/App.jsx

import React, { useState } from 'react';

function App() {
  const [question, setQuestion] = useState('');
  const [response, setResponse] = useState(null);
  const [loading, setLoading] = useState(false);

  const handleSubmit = async () => {
    setLoading(true);
    try {
      const res = await fetch('http://localhost:8000/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ question }),
      });
      const data = await res.json();
      setResponse(data);
    } catch (err) {
      console.error(err);
    } finally {
      setLoading(false);
    }
  };

  return (
    <div className="p-10 max-w-4xl mx-auto">
      <h1 className="text-2xl font-bold mb-4">Tax Compliance Copilot</h1>
      <textarea
        className="w-full p-2 border rounded"
        rows="3"
        value={question}
        onChange={(e) => setQuestion(e.target.value)}
        placeholder="Enter tax query..."
      />
      <button 
        onClick={handleSubmit} 
        className="mt-2 bg-blue-600 text-white px-4 py-2 rounded"
        disabled={loading}
      >
        {loading ? 'Analyzing...' : 'Get Advice'}
      </button>

      {response && (
        <div className="mt-6 p-4 border rounded bg-gray-50">
          <div className={`font-bold ${response.compliance_passed ? 'text-green-600' : 'text-red-600'}`}>
            Status: {response.compliance_passed ? 'Compliant' : 'Review Required'}
          </div>
          <p className="mt-2 whitespace-pre-wrap">{response.response}</p>
          
          {response.citations.length > 0 && (
            <div className="mt-4">
              <h3 className="font-semibold">Sources:</h3>
              <ul className="list-disc pl-5">
                {response.citations.map((cite, idx) => (
                  <li key={idx}>{cite.metadata.source} ({cite.metadata.article || cite.metadata.section})</li>
                ))}
              </ul>
            </div>
          )}
        </div>
      )}
    </div>
  );
}

export default App;

Running the Application

  1. Backend:

    cd backend
pip install -r requirements.txt
python -m uvicorn app.main:app --reload

  2. Ingest Data:

    curl -X POST http://localhost:8000/ingest

  3. Frontend:

    cd frontend
npm install
npm run dev

How We Evaluate: The "Correctness" Check

In this POC, evaluation is not an afterthought; it is a step in the workflow.

Enterprise Production Considerations

Limitations

Conclusion

Evaluating LLMs for domain correctness requires moving beyond surface-level fluency. By implementing a Multi-Agent LangGraph architecture with a dedicated Compliance Agent, we can automate the detection of hallucinations and ensure that every tax recommendation is grounded in verified sources. This POC demonstrates that with the right structure RAG for facts, Agents for logic, and State for context we can build AI systems that are not just smart, but also trustworthy and compliant.