Building Retrieval-Augmented Generation (RAG) in C#

RAG (Retrieval-Augmented Generation) makes a large language model answer from your own data. Before the model writes anything, the system searches a knowledge base of documents, wikis, tickets or product data, retrieves the most relevant passages and puts them into the prompt. The model then writes an answer grounded in that text, usually with citations.

It exists because an LLM only knows its training data. It doesn't know your internal documents, its knowledge has a cutoff, and it will make things up with total confidence. RAG addresses all three without retraining anything: update the documents and the answers change.

The term comes from a 2020 paper by Lewis et al., "Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks", which combined a retriever with a text generator so the model could look facts up rather than rely on what it had memorised. Today "RAG" means the practical pattern rather than that specific model: search first, then generate. It is what sits behind internal knowledge assistants, support bots, documentation search and every "chat with your PDFs" app.

What RAG gives you that a bare model can't

A model without RAG is sitting a closed-book exam. RAG hands it the right pages first. That changes more than accuracy. Answers can reflect documents written this morning, because retrieval happens at query time. They can draw on private data the model never saw. They can cite the passages they used, so users can check them. Keeping the system current means updating an index, which is cheap and immediate, rather than retraining a model.

The benefit people underrate is access control. Retrieval can filter by the user's permissions before the model sees a single word, which is something no amount of fine-tuning can do.

The architecture

A RAG system has two pipelines. Ingestion runs offline whenever documents change; the query pipeline runs for every question.

INGESTION (offline, when documents change)

Source documents (PDF, Word, HTML, wiki, database rows)
        ↓
Load and clean (extract text, strip boilerplate, keep metadata)
        ↓
Chunk (split into passages of a few hundred tokens, with overlap)
        ↓
Embed (embedding model turns each chunk into a vector)
        ↓
Store (vector + text + metadata in a vector store / search index)


QUERY (online, for every question)

User question
        ↓
(Optional) Rewrite query: fix typos, expand acronyms, add chat history
        ↓
Embed the question with the SAME embedding model
        ↓
Retrieve top-k chunks (vector search, often + keyword search)
        ↓
(Optional) Rerank: a cross-encoder reorders candidates by relevance
        ↓
Build prompt: instructions + top chunks + question
        ↓
LLM generates a grounded answer with citations
        ↓
Return answer + sources to the user

Ingestion: cleaning and chunking

Garbage in, garbage out applies to RAG more than almost anything else. Extract clean text, strip navigation and footers, and keep metadata such as title, URL, author, date, department and access level, because you'll need it for filtering and citations. Then actually look at your chunks. Tables, multi-column PDFs and scanned documents often extract as garbage, and no retriever can recover from that.

Chunking splits long documents into retrievable passages. Chunks that are too large dilute relevance and waste context; chunks that are too small lose their meaning. The main strategies are:

Fixed-size with overlap: simple and a sound baseline. The overlap preserves sentences that straddle a boundary.
Structure-aware: split on headings, sections, paragraphs or code blocks. Usually better for documentation.
Semantic: split where the topic changes, detected with embeddings.
Contextual: prepend a short summary of the document or section to each chunk, so it makes sense on its own.

There is no universally best chunk size. A few hundred tokens with 10 to 20 percent overlap, split on document structure where possible, is a common starting point. Then tune against an evaluation set.

Embeddings and the vector store

An embedding model turns text into a vector so that texts with similar meaning sit close together. "How do I reset my password?" and "I forgot my login credentials" land near each other despite sharing almost no words. Use the same embedding model for documents and queries, and if you ever switch models, re-embed everything.

A vector store indexes those embeddings for fast similarity search, usually with an approximate nearest neighbour algorithm such as HNSW. You can use a dedicated vector database, a search engine with vector support such as Azure AI Search or Elasticsearch, or a relational database with vector support, such as PostgreSQL with pgvector, or SQL Server and Azure SQL. My advice: if you already run a database that supports vectors, start there. A separate vector database earns its keep only when scale or specific features demand it.

Retrieval, reranking and generation

Retrieval returns the top-k chunks for the query, typically by cosine similarity. This is where metadata filters belong: tenant, language, product version and, above all, the user's permissions. Skip that last one and your RAG system becomes a data-leak tool.

A reranker, usually a cross-encoder, reads the query and each candidate together and scores relevance more accurately than vector distance can. The common pattern is to retrieve 20 to 50 candidates cheaply, rerank them, and keep the best 3 to 8. Keeping fewer matters: twenty mediocre chunks can bury the one good one.

Finally, the prompt tells the model to answer only from the supplied context, cite its sources, and say "I don't know" when the context doesn't contain the answer. That last instruction is what turns a demo into something you can trust. RAG doesn't eliminate hallucinations, since the model can still misread the context or answer beyond it, but grounding instructions plus faithfulness evaluation close most of the gap.

A minimal RAG app in C#

This console app indexes Markdown files from a docs folder, retrieves the most relevant chunks for each question and asks an LLM to answer with citations. It is built on Microsoft.Extensions.AI abstractions (IEmbeddingGenerator and IChatClient), so you can swap OpenAI for Azure OpenAI, Ollama or another provider without touching the RAG logic.

To keep every moving part visible, it stores vectors in a list and computes cosine similarity itself. In production, you'd replace the list with a real vector store.

Create the project

dotnet new console -n MiniRag
cd MiniRag
dotnet add package Microsoft.Extensions.AI.OpenAI --prerelease
dotnet add package System.Numerics.Tensors
mkdir docs

Put a few .md files in docs, and set two environment variables: OPENAI_API_KEY and CHAT_MODEL, the chat model you want to use.

Program.cs

using System.ClientModel;
using System.Numerics.Tensors;
using System.Text;
using Microsoft.Extensions.AI;
using OpenAI;

var openAi = new OpenAIClient(new ApiKeyCredential(
    Environment.GetEnvironmentVariable("OPENAI_API_KEY")!));

IEmbeddingGenerator<string, Embedding<float>> embedder =
    openAi.GetEmbeddingClient("text-embedding-3-small").AsIEmbeddingGenerator();

IChatClient chat =
    openAi.GetChatClient(Environment.GetEnvironmentVariable("CHAT_MODEL")!).AsIChatClient();

// ---------- INGESTION ----------
var chunks = new List<Chunk>();
foreach (string file in Directory.GetFiles("docs", "*.md"))
{
    List<string> pieces = ChunkText(File.ReadAllText(file), maxChars: 1200, overlapChars: 200).ToList();
    var embeddings = await embedder.GenerateAsync(pieces);   // one batched call per file

    for (int i = 0; i < pieces.Count; i++)
        chunks.Add(new Chunk(Path.GetFileName(file), pieces[i], embeddings[i].Vector));
}
Console.WriteLine($"Indexed {chunks.Count} chunks.");

// ---------- QUERY ----------
while (true)
{
    Console.Write("\nQuestion: ");
    string? question = Console.ReadLine();
    if (string.IsNullOrWhiteSpace(question)) break;

    ReadOnlyMemory<float> q = await embedder.GenerateVectorAsync(question);

    var top = chunks
        .Select(c => (Chunk: c, Score: TensorPrimitives.CosineSimilarity(q.Span, c.Vector.Span)))
        .OrderByDescending(x => x.Score)
        .Take(4)
        .ToList();

    string context = string.Join("\n\n", top.Select((x, i) =>
        $"[{i + 1}] (source: {x.Chunk.Source})\n{x.Chunk.Text}"));

    ChatResponse response = await chat.GetResponseAsync(
    [
        new(ChatRole.System,
            "Answer ONLY from the numbered context. Cite sources like [1]. " +
            "If the context does not contain the answer, say you don't know."),
        new(ChatRole.User, $"Context:\n{context}\n\nQuestion: {question}")
    ]);

    Console.WriteLine(response.Text);
    foreach (var (chunk, score) in top)
        Console.WriteLine($"  - {chunk.Source} (score {score:F3})");
}

// Paragraph-aware chunking with a character overlap
static IEnumerable<string> ChunkText(string text, int maxChars, int overlapChars)
{
    string[] paragraphs = text.Replace("\r\n", "\n")
        .Split("\n\n", StringSplitOptions.RemoveEmptyEntries | StringSplitOptions.TrimEntries);

    var current = new StringBuilder();
    foreach (string p in paragraphs)
    {
        if (current.Length > 0 && current.Length + p.Length > maxChars)
        {
            string done = current.ToString();
            yield return done;
            string tail = done[^Math.Min(overlapChars, done.Length)..];
            current.Clear().Append(tail).Append("\n\n");
        }
        current.Append(p).Append("\n\n");
    }
    if (current.Length > 0) yield return current.ToString();
}

record Chunk(string Source, string Text, ReadOnlyMemory<float> Vector);

Run it, then break it on purpose

dotnet run

Ask a question your documents answer, then one they don't. The second answer should be "I don't know". If the model answers anyway, tighten the system prompt before you change anything else.

Moving to a real vector store

The list is fine for a few thousand chunks. Beyond that, use Microsoft.Extensions.VectorData, which puts one API over many stores. You describe the record with attributes, then call UpsertAsync and SearchAsync:

using Microsoft.Extensions.VectorData;

internal class DocChunk
{
    [VectorStoreKey]
    public string Id { get; set; } = "";

    [VectorStoreData]
    public string Source { get; set; } = "";

    [VectorStoreData]
    public string Text { get; set; } = "";

    // text-embedding-3-small returns 1536 dimensions by default
    [VectorStoreVector(1536, DistanceFunction = DistanceFunction.CosineSimilarity)]
    public ReadOnlyMemory<float> Vector { get; set; }
}

Microsoft's vector search quickstart shows the full flow with an in-memory store, and the same code targets production connectors. For an Azure-scale design, see Building Enterprise-Grade RAG on Azure.

Making retrieval better

Once the basic loop works, almost every quality problem traces back to retrieval. These are the upgrades worth knowing, roughly in the order I'd reach for them.

Hybrid search and reranking

Vector search is good at meaning and weak at exact terms: error codes, product SKUs, names and acronyms. Keyword search (BM25) is the opposite. Hybrid search runs both and merges the results, commonly with Reciprocal Rank Fusion (RRF). For most enterprise content, hybrid search plus a reranker beats either method alone.

Anthropic's contextual retrieval study from September 2024 puts numbers on this. Adding a short, generated context to each chunk before embedding cut top-20 retrieval failures by 35 percent. Combining that with contextual BM25 cut them by 49 percent, and adding reranking brought the reduction to 67 percent.

Query transformation

Users don't write good search queries, so rewrite them. In a chat, "what about the second one?" needs turning into a standalone question using the conversation history. Multi-query generates several phrasings and merges the results. HyDE has the model write a hypothetical answer and searches with that answer's embedding.

GraphRAG for corpus-wide questions

GraphRAG, from Microsoft Research, extracts entities and relationships from your documents into a knowledge graph, groups them into communities and writes a summary of each. Its global search uses those summaries to answer broad questions such as "What are the main themes across all incident reports?", which plain chunk retrieval answers poorly. Its local search fans out from specific entities. The price is cost and complexity, since building the graph takes many LLM calls at indexing time. Use it when questions span the whole corpus or hinge on relationships.

Agentic RAG for multi-step questions

In agentic RAG, retrieval becomes a tool the model calls rather than a fixed first step. The agent decides whether to search at all, which index or API to query, and whether to refine the query and search again after reading the results.

Question
        ↓
Agent plans: "I need the refund policy AND this customer's order"
        ↓
Tool call: search_policies("refund window digital goods")
        ↓
Tool call: get_order(orderId)
        ↓
Agent checks: enough evidence? If not, search again
        ↓
Grounded answer with citations

It handles multi-step questions far better, at the cost of more latency, more tokens and harder debugging. Exposing retrieval as an MCP tool is a common way to build it.

Evaluating a RAG system

Evaluate retrieval and generation separately. If retrieval misses the right chunk, no model will produce the right answer, and no amount of prompt tuning will fix what is really a search problem.

Stage	Metric	Question it answers
Retrieval	Recall@k / hit rate	Is the correct chunk in the top k?
Retrieval	MRR / nDCG	How high is the correct chunk ranked?
Retrieval	Context precision	How much of the retrieved context is relevant?
Generation	Faithfulness / groundedness	Is every claim supported by the context?
Generation	Answer relevancy	Does the answer address the question?
End to end	Correctness vs a reference answer	Is the answer actually right?
Operations	Latency, cost per query, "I don't know" rate	Can you afford and trust it in production?

The loop that works is unglamorous:

Build a test set of 50 to 200 real questions, each with the expected answer and the source passage.
Measure retrieval recall@k on its own first.
Score faithfulness and relevancy with an LLM-as-judge framework such as Ragas, and spot-check by hand.
Change one thing at a time (chunk size, hybrid search, reranker, prompt) and re-run.
Run the suite in CI so regressions show up before users see them.

Without a test set, every change is a guess. How to Build a RAG Evaluation Pipeline With Python walks through a complete pipeline.

RAG, fine-tuning or long context?

Aspect	RAG	Fine-tuning	Long context
What it changes	What the model can see at query time	The model's weights and behaviour	How much you paste into one prompt
Best for	Large, changing, private knowledge with citations	Style, format, domain language, narrow tasks	Small or one-off document sets
Freshness	Update the index any time	Retrain to update	Always current (you send it each time)
Citations	Natural	Hard	Possible
Cost profile	Indexing plus small prompts	Training cost up front	Large prompt on every call (caching helps)
Access control	Filter per user at retrieval	Not possible once trained in	You choose what to include

Long context deserves a fair hearing. Several current frontier models accept a 1-million-token context window, which makes "just put it all in the prompt" a real option for small corpora. Anthropic's own guidance in the contextual retrieval post is that a knowledge base under about 200,000 tokens can often go straight into the prompt, with prompt caching to keep the cost down. For large, changing or permissioned knowledge, though, RAG is still cheaper and faster per query, and it gives you control over exactly what the model sees.

My rule of thumb: long context for small sets, RAG for large or changing knowledge, and fine-tuning for behaviour rather than facts. They also combine well, and a fine-tuned or carefully prompted model on top of RAG is a common production setup.

Before you ship

Two risks don't show up in a demo. The first is prompt injection through your own documents: retrieved text can contain instructions, so treat it strictly as data and never let retrieved content trigger privileged actions on its own. The second is a stale index. Re-index when documents change, and delete the vectors for documents you remove, or the assistant will keep confidently quoting a policy you retired.