Converting Messy Documents to Markdown in Enterprise RAG

If you have built an enterprise Retrieval-Augmented Generation (RAG) system, you already know the dirty secret of the industry: The LLM is rarely the bottleneck; the data ingestion pipeline is. Enterprises do not store their knowledge in neat, pre-formatted JSON files. Knowledge is trapped in 50-page PDFs with complex tables, nested DOCX onboarding guides, and legacy TXT configuration logs. If you naively extract text from these files and chunk them using a simple character-count splitter, your RAG system will suffer from "context fragmentation." The LLM will receive a chunk of text without knowing what document it came from, what chapter it belongs to, or what the parent topic was.

The solution? The Markdown Intermediate Representation. In this end-to-end guide, we will build an enterprise-grade LangChain ingestion pipeline that converts PDFs, DOCX, and TXT files into a unified Markdown format. We will then use LangChain’s MarkdownHeaderTextSplitter to preserve semantic hierarchy, ensuring your RAG application always has the full context.

The Real-World Use Case: GlobalCorp IT & HR Helpdesk

The Scenario: GlobalCorp has 10,000 employees. The IT and HR departments are drowning in support tickets. Employees are constantly asking:

"What is the maximum reimbursement limit for international travel?" (Found in a 40-page HR Policy PDF)
"How do I configure the Cisco AnyConnect VPN on macOS?" (Found in an IT SOP DOCX)
"What is the error code 404-B meaning in the legacy billing system?" (Found in a plain TXT log dictionary)

The Goal: Build an internal RAG chatbot that accurately answers these questions, citing the exact document and section.

The Challenge: HR policies are heavy on tables. IT guides are heavy on nested headers (H1 -> H2 -> H3). TXT files are unstructured. We need a unified ingestion strategy.

Why Markdown is the "Silver Bullet" for RAG

Before writing code, we must understand why we are converting everything to Markdown.

Structural Preservation: Markdown natively supports headers (#, ##), tables, and lists.
Metadata Enrichment: When we split Markdown by headers, we can attach the "Header Path" (e.g., HR Policy > Travel > International Flights) as metadata to every single chunk.
Table Readability: LLMs understand Markdown tables significantly better than raw CSV or extracted PDF table gibberish.
Universal Format: It acts as a normalized intermediate layer. Whether the source is a PDF or a DOCX, the downstream chunking and embedding logic only needs to understand one format.

The Architecture

Our LangChain pipeline will follow these steps:

Universal Conversion: Use Microsoft’s markitdown (the 2025/2026 enterprise standard for LLM document parsing) to convert PDF, DOCX, and TXT to Markdown.
Semantic Chunking: Use LangChain’s MarkdownHeaderTextSplitter to split documents based on H1, H2, and H3 tags.
Vectorization: Embed the chunks using OpenAI and store them in Chroma (with enterprise metadata filtering).
Retrieval & Generation: Query the vector store and generate an answer with source citations.

Step-by-Step Implementation

Prerequisites

Install the required libraries. We are using markitdown for robust document-to-markdown conversion, and the latest LangChain ecosystem.

pip install langchain langchain-openai langchain-chroma langchain-text-splitters markitdown python-dotenv

Step 1: The Universal Markdown Loader

LangChain has native loaders for PDF and DOCX, but they output raw text. We will create a custom LangChain BaseLoader that wraps markitdown to output pure Markdown, preserving tables and structures.

import os
from pathlib import Path
from typing import Iterator
from markitdown import MarkItDown
from langchain_core.document_loaders import BaseLoader
from langchain_core.documents import Document

class EnterpriseMarkdownLoader(BaseLoader):
    """
    Custom LangChain Loader that converts PDF, DOCX, and TXT 
    into Markdown using Microsoft's MarkItDown.
    """
    def __init__(self, file_path: str):
        self.file_path = file_path

    def lazy_load(self) -> Iterator[Document]:
        md = MarkItDown()
        file_ext = Path(self.file_path).suffix.lower()
        
        # MarkItDown handles pdf, docx, txt, and more natively
        result = md.convert(self.file_path)
        
        yield Document(
            page_content=result.text_content,
            metadata={
                "source": os.path.basename(self.file_path),
                "file_type": file_ext,
                "full_path": self.file_path
            }
        )

Step 2: Ingestion & Semantic Chunking

Now, we ingest the documents. The magic happens here: instead of splitting by 1000 characters, we split by Headers.

from langchain_text_splitters import MarkdownHeaderTextSplitter, RecursiveCharacterTextSplitter
from langchain_core.documents import Document

def process_documents_to_chunks(directory_path: str) -> list[Document]:
    headers_to_split_on = [
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3"),
    ]
    md_splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers_to_split_on)
    
    # Fallback for TXT files or docs without headers
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
    
    all_chunks = []
    
    for file_name in os.listdir(directory_path):
        if file_name.endswith(('.pdf', '.docx', '.txt')):
            file_path = os.path.join(directory_path, file_name)
            loader = EnterpriseMarkdownLoader(file_path)
            docs = loader.load()
            
            for doc in docs:
                # Attempt to split by Markdown headers
                md_chunks = md_splitter.split_text(doc.page_content)
                
                for chunk in md_chunks:
                    # Merge original metadata (source, file_type) with header metadata
                    chunk.metadata.update(doc.metadata)
                    
                # If the document had no headers (common in TXT files), fallback to character split
                if not md_chunks:
                    fallback_chunks = text_splitter.split_documents([doc])
                    all_chunks.extend(fallback_chunks)
                else:
                    all_chunks.extend(md_chunks)
                    
    print(f"Successfully generated {len(all_chunks)} semantic chunks.")
    return all_chunks

Notice how chunk.metadata now contains keys like Header 1, Header 2, etc. If a chunk comes from the "VPN Setup" section, the metadata will literally say {"Header 1": "IT Policies", "Header 2": "VPN Setup"}.

Step 3: Vector Store Integration

We will use ChromaDB for this example, but in an enterprise setting, you would swap this for Pinecone, Milvus, or pgvector.

from langchain_openai import OpenAIEmbeddings
from langchain_chroma import Chroma

def create_vector_store(chunks: list[Document], persist_directory: str = "./chroma_db"):
    embeddings = OpenAIEmbeddings(model="text-embedding-3-large")
    
    # Chroma will automatically index the metadata for filtering later!
    vectorstore = Chroma.from_documents(
        documents=chunks,
        embedding=embeddings,
        persist_directory=persist_directory,
        collection_name="globalcorp_knowledge"
    )
    return vectorstore

Step 4: Building the RAG Chain

Finally, we build the retrieval chain. Because we preserved the headers in our metadata, we can instruct the LLM to format the citations beautifully.

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.runnables import RunnablePassthrough
from langchain_core.output_parsers import StrOutputParser

def format_docs_with_metadata(docs):
    formatted = []
    for i, doc in enumerate(docs):
        # Reconstruct the "breadcrumb" trail from metadata
        breadcrumb = " > ".join([doc.metadata.get(f"Header {i}", "") for i in range(1, 4) if doc.metadata.get(f"Header {i}")])
        source = doc.metadata.get("source", "Unknown")
        
        context_block = f"""
        --- Source: {source} | Context Path: {breadcrumb} ---
        {doc.page_content}
        """
        formatted.append(context_block)
    return "\n".join(formatted)

prompt = ChatPromptTemplate.from_template("""
You are the GlobalCorp Enterprise Assistant. 
Answer the user's question based ONLY on the provided context. 
If the answer is not in the context, state that you do not have enough information.
Always cite the Source and Context Path in your answer.

Context:
{context}

Question: {question}
""")

def build_rag_chain(vectorstore):
    llm = ChatOpenAI(model="gpt-4o", temperature=0)
    retriever = vectorstore.as_retriever(search_kwargs={"k": 4})
    
    rag_chain = (
        {
            "context": retriever | format_docs_with_metadata,
            "question": RunnablePassthrough()
        }
        | prompt
        | llm
        | StrOutputParser()
    )
    return rag_chain

Step 5: Execution

Let's tie it all together and query the system.

if __name__ == "__main__":
    # 1. Ingest and Chunk
    DATA_DIR = "./enterprise_docs" # Assume this contains policy.pdf, it_sop.docx, logs.txt
    chunks = process_documents_to_chunks(DATA_DIR)
    
    # 2. Vectorize
    vectorstore = create_vector_store(chunks)
    
    # 3. Query
    rag_chain = build_rag_chain(vectorstore)
    
    query = "How do I set up my VPN on a Mac, and what is the HR policy on working remotely while traveling?"
    response = rag_chain.invoke(query)
    
    print("\n--- ASSISTANT RESPONSE ---")
    print(response)

Enterprise Best Practices & Gotchas

When moving this from a local prototype to a production enterprise environment, keep these rules in mind:

1. The "TXT File" Edge Case

Plain text files (like system logs or legacy configs) rarely contain Markdown headers (#). The MarkdownHeaderTextSplitter will ignore them and return the whole file as one chunk.
The Fix: As implemented in Step 2, always chain a RecursiveCharacterTextSplitter as a fallback for documents that yield zero header-based splits.

2. Metadata Filtering is Your Best Friend

Because we converted to Markdown and extracted headers into metadata, you can now use Self-Querying Retrievers.
If an employee asks, "What is the IT policy for...", the LLM can automatically filter the vector search to file_type == ".docx" or department == "IT", drastically reducing hallucinations and retrieval latency.

3. Handling Complex PDF Tables

While markitdown is excellent at converting PDF tables to Markdown tables, highly complex, multi-span PDF tables can still break. For mission-critical financial documents, consider routing specific PDFs through an LLM-vision model (like GPT-4o) to manually reconstruct the Markdown table before ingestion.

4. Evaluation (RAGAS / LangSmith)

Never deploy an ingestion pipeline without evaluating it. Use LangSmith to trace your runs. Specifically, measure the Context Recall. If the MarkdownHeaderTextSplitter is cutting a paragraph away from its defining H2 header, your context recall will drop. Tweak the headers_to_split_on array based on your specific document layouts.

Conclusion

The shift from "raw text extraction" to "Markdown Intermediate Representation" is one of the highest-ROI changes you can make to an enterprise RAG pipeline. By leveraging tools like markitdown and LangChain’s MarkdownHeaderTextSplitter, you stop treating documents as dumb strings of text, and start treating them as structured, hierarchical knowledge graphs.

Your LLM doesn't just get the answer anymore; it gets the context required to trust it.