from langchain.text_splitter import RecursiveCharacterTextSplitter # Example: Recursive chunking with overlap text_splitter = RecursiveCharacterTextSplitter( chunk_size=500, # tokens per chunk chunk_overlap=50, # overlap to preserve context separators=["\n\n", "\n", " ", ""] ) chunks = text_splitter.split_text(long_document) print(f"Created {len(chunks)} chunks") Chunk size : Typically 300–1000 tokens. Overlap : 10–20% of chunk size helps preserve context across boundaries. Storage : Chunks are embedded and stored in a vector DB (e.g., FAISS, Pinecone).