Retrieval-Augmented Generation (RAG) is the pattern behind almost every "chat with your documents" product, enterprise knowledge base, and AI-powered support system. The concept is elegant: instead of training a model on your data, you retrieve relevant chunks at query time and include them in the prompt as context.
The naive implementation takes about two hours to build and works reasonably well in demos. Making it reliable enough for production use on real data is a different engineering challenge entirely.
Why Naive RAG Breaks
The standard RAG pipeline goes: embed query → find top-k similar chunks by cosine similarity → stuff chunks into prompt → generate answer.
This breaks in predictable ways:
The retrieved chunks don't contain the answer. Similarity search finds text that looks like the query, not text that answers it. A question about "contract termination penalties" might retrieve chunks about "contract signing procedures" because they share vocabulary.
The retrieved chunks are incomplete. A policy document might spread relevant information across three sections. Retrieving any one chunk gives a partial answer. The LLM then confidently answers with incomplete information.
The chunks conflict. In large document sets with multiple versions or contradicting sources, the top-k chunks might contain conflicting information. Without handling this, the LLM blends them or picks one arbitrarily.
The query is ambiguous. "What is the refund policy?" could mean the standard policy, the policy for a specific product, or the policy in a specific jurisdiction. Embedding-based retrieval has no way to disambiguate without more context.
The Production RAG Stack
Here's what a production-grade RAG system actually looks like:
1. Better chunking
Naive chunking splits documents every N tokens. This breaks semantic units — a sentence that starts a chunk might need the previous sentence for context.
Better approaches:
- Semantic chunking: split at logical boundaries (paragraph breaks, section headers, numbered list items) rather than token count
- Hierarchical chunking: store chunks at multiple granularities — sentence, paragraph, section — and retrieve at the appropriate level
- Sliding window with overlap: each chunk overlaps with the previous and next chunk, so context isn't lost at boundaries
For structured documents (contracts, policies, technical specs), extract the document structure explicitly. A clause in a contract is a more meaningful unit than 512 tokens of text.
2. Hybrid search
Pure vector search misses exact matches. If a user asks "what is clause 8.3.2?", vector search will find semantically similar clauses but may not find clause 8.3.2 specifically.
Production RAG systems combine:
- Dense retrieval (vector search): captures semantic similarity
- Sparse retrieval (BM25/keyword): captures exact term matches
- Re-ranking: a smaller model that re-scores the combined results for relevance
The combination significantly outperforms either approach alone on heterogeneous queries.
3. Query understanding and rewriting
Before retrieval, process the query:
async function prepareQuery(userQuery: string) {
// Expand the query with synonyms and related terms
const expanded = await expandQuery(userQuery);
// If conversational context exists, resolve pronouns/references
const resolved = await resolveReferences(userQuery, conversationHistory);
// Generate multiple phrasings for multi-query retrieval
const variants = await generateQueryVariants(resolved);
return { primary: resolved, variants, expanded };
}
Multi-query retrieval — generating 3–5 phrasings of the same question and retrieving chunks for all of them — dramatically improves recall on complex questions.
4. Context assembly and deduplication
After retrieval, before building the prompt:
- Deduplicate: if multiple query variants return the same chunk, include it once
- Order by relevance, not retrieval order
- Include source metadata (document name, section, date) so the LLM can reason about recency and authority
- Respect token budget: prioritise highest-relevance chunks, truncate gracefully
5. Grounded generation
Instruct the model explicitly to:
- Answer only from the provided context
- Cite the specific source for each claim
- Say "I don't have information about this in the provided documents" rather than hallucinating
System: You are a knowledge assistant. Answer questions using ONLY the provided context.
For each claim, cite the source document and section. If the context doesn't contain
sufficient information to answer, say so explicitly — do not infer or guess.
Context:
[Document: Employee Handbook v4.2, Section 8 — Leave Policy]
Annual leave entitlement is 25 days per calendar year for permanent employees...
[Document: HR Policy Update 2026-03, Section 2]
Effective January 2026, annual leave entitlement increases to 28 days...
The explicit instruction to cite sources serves two purposes: it forces the model to ground its answer in the retrieved text, and it gives users the ability to verify the answer.
6. Confidence and coverage signals
Add signals to every RAG response:
- Retrieval confidence: average cosine similarity of top-k chunks. Below 0.7 on a normalised scale, the retrieval was weak — surface this to users.
- Coverage: did the retrieved chunks actually address the question? A second LLM call to evaluate this is cheaper than user trust erosion from wrong answers.
- Source freshness: if retrieved chunks are from documents older than a threshold, flag this.
Evaluating RAG Quality
The most important investment in a RAG system is evaluation infrastructure. Without it, you cannot measure improvements.
Build a golden dataset: 100–200 question/answer pairs with known correct answers drawn from your documents. Run your RAG pipeline against them and measure:
- Faithfulness: is the answer supported by the retrieved context?
- Answer relevance: does the answer actually address the question?
- Context recall: did the retrieval find the chunks needed to answer correctly?
Tools like RAGAS make this straightforward to implement. Run your eval suite on every significant change to chunking strategy, retrieval parameters, or prompt.
When RAG Is the Wrong Tool
RAG is not the right answer for every knowledge access problem:
- Small, stable knowledge bases (< 50 documents that rarely change): fine-tune or use full-context models
- Highly structured queries (show me all contracts expiring in Q3): SQL over a relational database, not vector search
- Real-time data (current stock prices, live inventory): RAG retrieves from indexed documents, not live systems — use tool calling instead
RAG shines for large, semi-structured knowledge bases with heterogeneous content that updates regularly and needs to be queried in natural language.
If you're building a knowledge system and want to get the architecture right from the start, get in touch.