Building a RAG Pipeline: From Documents to Grounded Answers🔗

Ask a language model a question about your own documents and it will happily answer — often confidently, sometimes correctly, occasionally inventing a source that never existed. The model wasn't trained on your notes, your product docs, or last week's meeting minutes, so it fills the gap the only way it knows how: by guessing in fluent prose.

Retrieval-augmented generation, or RAG, is the fix. The idea is almost embarrassingly simple: before the model answers, go find the relevant text and hand it over. The model stops guessing and starts reading. But the gap between "almost embarrassingly simple" and "actually reliable" is where all the real work lives, and that gap is what this post is about.

Here's the pipeline I keep coming back to, and the decisions that matter at each stage.

The shape of a RAG system🔗

Every RAG system, no matter how elaborate, is the same five movements:

1. Ingest — collect the source documents.
2. Chunk — split them into passages small enough to retrieve.
3. Embed — turn each chunk into a vector and store it.
4. Retrieve — at query time, find the chunks closest to the question.
5. Generate — hand those chunks to the model and ask it to answer from them.

The first three happen offline, once, when your data changes. The last two happen on every request. Getting this split right is the first design decision: indexing is expensive and can be batched; retrieval and generation are on the critical path and need to be fast.

Chunking is where quality is won or lost🔗

The instinct is to skip past chunking as plumbing. Resist it. Chunking is the single highest-leverage decision in the whole pipeline, because it decides what the retriever is even able to find.

Chunk too large and each passage carries several ideas at once; the embedding becomes a blurry average and retrieval gets vague. Chunk too small and you sever the context a sentence needs to make sense — a paragraph that says "this changed in the latest release" is useless without knowing what this is.

A few things that have served me well:

• Split on structure, not character counts. Respect headings, paragraphs, and list boundaries. A chunk that ends mid-sentence is a chunk that retrieves badly.
• Overlap a little. Carrying 10–15% of the previous chunk into the next one keeps ideas that straddle a boundary retrievable from either side.
• Keep the breadcrumbs. Prepend the document title and section heading to each chunk before embedding. "Refund policy › Exceptions" is a very different vector than the bare paragraph, and it costs you almost nothing.

Tune chunking against real questions, not vibes. If you can't retrieve the right passage for a query you know is answered in the corpus, no amount of prompt engineering downstream will save you.

Embeddings: turning meaning into geometry🔗

An embedding model maps text into a high-dimensional vector so that passages with similar meaning land near each other. "How do I cancel my subscription?" and "steps to end your plan" share almost no words but should sit close together in that space. That's the whole trick — semantic proximity becomes geometric proximity, and geometric proximity is something a computer can search fast.

Two practical notes here. First, use the same embedding model for your documents and your queries. They have to live in the same space or the distances are meaningless. Second, the token budget of your embedding model is a hard constraint on chunk size — another reason chunking and embedding are really one decision, not two. If you've read my tokeniser post, the same counting discipline applies: know how many tokens your chunks actually consume before you're surprised by a truncation.

Storage and retrieval: the vector database🔗

Once every chunk is a vector, you need somewhere to keep them and a way to find the nearest ones to a query vector. That's a vector database. At small scale you can brute-force it — compare the query against every stored vector — but as the corpus grows you'll want an approximate nearest-neighbour index (HNSW is the common one) that trades a sliver of accuracy for a massive speed-up.

Retrieval quality lives and dies on two ideas most demos skip:

• Hybrid search. Pure vector search is great at meaning but bad at exact tokens — product codes, error numbers, proper nouns. Pairing it with old-fashioned keyword search (BM25) and blending the scores catches what each misses alone.
• Reranking. Retrieve generously — say the top 20 candidates — then run a cross-encoder reranker to reorder them by true relevance and keep the best 4 or 5. The first pass optimises for recall (don't miss the answer); the rerank optimises for precision (don't drown it in noise). This two-stage move is the cheapest large win available.

Generation: grounding, not decoration🔗

Now the part everyone pictures when they hear "RAG": you take the top chunks, drop them into the prompt, and ask the model to answer. The prompt is where you turn retrieved text into a constraint rather than a suggestion.

Be explicit and be strict:

Answer the question using ONLY the context below.
If the answer isn't in the context, say you don't know.
Cite the source of each claim.

Context:
{retrieved_chunks}

Question: {user_question}

That "say you don't know" line is doing heavy lifting. The entire point of RAG is to trade confident fabrication for honest grounding, and you have to ask for that trade — the model's default is to please you, not to abstain. Requiring citations does double duty: it nudges the model to stay anchored to the text, and it gives your users a way to verify the answer instead of trusting it blind.

The failures you'll actually hit🔗

A demo works on the first three questions. A system has to survive the next three hundred. The failures cluster:

• Retrieval missed. The answer exists in the corpus but the right chunk never surfaced. Almost always a chunking or hybrid-search problem, not a model problem. Fix it upstream.
• Retrieved but ignored. The right chunk was in the prompt and the model answered from its own memory anyway. Tighten the instruction, and put the context close to the question — models weight recent tokens.
• Stale index. The documents changed; the vectors didn't. Your pipeline needs a reindexing story from day one, not as an afterthought.
• Context overflow. You retrieved so much that the important passage got truncated or buried. More context is not more better — retrieve less, but retrieve right.

Notice how many of these are retrieval problems wearing a generation costume. When a RAG answer is wrong, the reflex is to blame the model or rewrite the prompt. Look at what was retrieved first. Nine times out of ten the fix is earlier in the pipeline than you think.

Start simple, then earn the complexity🔗

You do not need a reranker, hybrid search, and a query-rewriting agent to ship your first version. You need clean chunks, a decent embedding model, top-k retrieval, and a strict prompt. Get that honest baseline working end to end, measure where it actually fails on real questions, and add machinery only where the numbers tell you to.

RAG isn't magic and it isn't a product you buy — it's a pipeline you tune. The models will keep getting better underneath you, but the discipline stays the same: give the model the right text, ask it to stay grounded, and make it easy for a human to check the work. Do that, and you turn a confident guesser into something you can actually trust.