RAG that actually works in production

Most RAG demos work because the demo is rigged. Three clean documents, a question that maps obviously to one of them, and a slick answer that makes everyone in the room nod. Then you point the same pipeline at ten thousand real documents with inconsistent formatting and overlapping content, and it falls apart. Getting RAG in production to actually work is mostly a fight against that gap, and very little of the fight is about the language model.

I have built retrieval systems that had to hold up under real query load, and the lesson keeps repeating: your answer quality is capped by your retrieval quality. A great model fed bad chunks gives you a confident wrong answer, which is worse than no answer at all.

Retrieval quality is the whole game

If the right context is not in the chunks you pulled, the model cannot magic it into existence. So before tuning prompts or swapping models, I spend almost all my early effort measuring whether retrieval is even surfacing the correct documents. Build a set of real questions with known good source passages, then check how often those passages show up in the top results. If that number is low, nothing downstream matters.

Plain vector search alone is rarely enough. Pure semantic similarity misses exact matches like error codes, product names, and IDs that a user types verbatim. Combining keyword search with vector search, then reranking the merged set, is the single change that moves the needle most for me. It is less glamorous than a bigger model and far more effective.

Chunking is a design decision, not a default

The naive approach, splitting everything into fixed 500 token blocks, throws away structure that the document was carrying for free. A heading tells you what a section is about. A table row means nothing once you slice it in half. I chunk along the natural boundaries of the content, keep enough overlap that an idea is not cut mid-sentence, and attach metadata like the source title and section so retrieval has more to work with.

There is no universal chunk size. Dense technical docs want smaller, tighter chunks. Narrative content wants larger ones that keep context together. The only way to know is to measure retrieval against your own questions, which brings me to the part everyone skips.

You need evals or you are flying blind

Without evaluation you are tuning by vibes, and vibes lie. I keep a fixed set of representative questions with expected answers and known sources, and I run the whole pipeline against it every time I change anything. Two numbers matter most:

• Retrieval hit rate: how often the correct source shows up in the results you actually pass to the model
• Answer faithfulness: whether the generated answer is supported by the retrieved context or quietly made up

When I change the chunking strategy or the reranker and the hit rate drops, I know immediately, before it reaches a single user. That feedback loop is the entire difference between a system that improves and one that drifts.

Cite sources and make hallucination visible

A production RAG system should show its work. Every answer should point back to the chunks it used, both because users need to verify and because it makes failures debuggable. When a wrong answer comes in, the citation tells me instantly whether retrieval pulled the wrong thing or the model ignored the right thing. Those are completely different bugs with completely different fixes.

I also instruct the model to say it does not know when the context does not contain the answer. That sounds obvious and almost nobody does it. A system that confidently invents an answer when retrieval fails is far more dangerous than one that admits the gap, especially once non-technical people start trusting it.

The boring plumbing decides reliability

The parts nobody tweets about are the parts that keep a RAG system alive. Re-indexing when source documents change, so answers are not stale. Handling documents that fail to parse instead of silently dropping them. Caching embeddings so you are not paying to recompute the same vectors. Rate limiting and retries around the embedding and model calls so a provider hiccup does not take the feature down.

None of this is exciting and all of it is the difference between a demo and a system people depend on. The model is a small piece. Treat RAG as an information retrieval problem with a language model stapled to the end, measure relentlessly, and show your sources. Do that and it holds up. Skip it and you ship a very expensive way to be confidently wrong.