RAG in production: what actually breaks

Every RAG demo works. You load fifty documents, ask three questions you already know the answers to, and the model responds beautifully. Then you point it at 200,000 documents and real users, and the quality falls off a cliff.

The failure is almost never the language model. It is retrieval. If the right chunk never reaches the context window, no amount of prompt engineering will save the answer — the model will simply write something plausible instead. That is the whole problem in one sentence.

1. Chunking decided your ceiling before you started

Fixed-size chunking — split every 1,000 characters — is the default in every tutorial and it is the single biggest cause of bad retrieval. It cuts tables in half, separates a heading from the paragraph it introduces, and strands a pronoun three chunks away from its referent.

What works better, in rough order of effort:

• Split on structure first. Headings, list boundaries, table rows. Your documents already have semantics — use them before falling back to character counts.
• Overlap deliberately. 10–15% overlap recovers most boundary losses. More than that and you inflate your index and start returning near-duplicates.
• Attach context to every chunk. Prepend the document title and heading path. A chunk that reads Refund policy > EU > Timeframes retrieves far better than the same paragraph naked.
• Keep small tables whole. Splitting a table destroys it. If it does not fit, summarise it into the chunk and store the full table as metadata.

2. Pure vector search is not enough

Dense embeddings are excellent at meaning and surprisingly bad at exact tokens. Ask for error code ERR_4021 or a part number and a pure vector search will happily return five semantically similar paragraphs that mention none of them.

Hybrid retrieval fixes this: run BM25 keyword search and dense vector search in parallel, then fuse the two ranked lists. Reciprocal Rank Fusion is about six lines of code and consistently beats either method alone.

def reciprocal_rank_fusion(rankings, k=60):
    """Fuse several ranked lists of doc ids into one."""
    scores = {}
    for ranking in rankings:
        for rank, doc_id in enumerate(ranking):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1)
    return sorted(scores, key=scores.get, reverse=True)

fused = reciprocal_rank_fusion([bm25_hits, vector_hits])

3. Top-k is doing two jobs badly

Retrieval wants high recall — cast a wide net so the right chunk is somewhere in the results. Generation wants high precision — give the model few enough chunks that the answer is not buried in noise. A single top-k value cannot serve both.

Split the stage in two. Retrieve 50 candidates for recall, then rerank with a cross-encoder and pass the best 5 to the model. A cross-encoder reads the query and the chunk together instead of comparing two pre-computed vectors, so it is far more accurate — and because it only scores 50 candidates, the cost stays manageable.

4. You have no idea whether changes help

Most teams tune RAG by asking it a few questions and forming an impression. That is not measurement. You need a golden set: 100–200 real questions with the chunk IDs that should be retrieved for each.

With that in place you can measure recall@k — how often the correct chunk appears in the top k — before the model is involved at all. Retrieval is now a search problem with a number attached, and you can tune chunk size, overlap, embedding model and fusion weights against it in minutes instead of arguing about vibes.

5. The model answers when it should refuse

When retrieval returns nothing relevant, an unconstrained model will still produce a fluent, confident, wrong answer. That is worse than silence, because a plausible fabrication is harder to catch than an obvious blank.

Two guardrails handle most of this. First, threshold on the reranker score and refuse below it. Second, require citations: instruct the model to quote the span it used, then verify that span actually exists in the retrieved text. If it does not, the answer is not grounded and should not ship.

Where to start

1. 1Build a golden set of real questions. Nothing else can be measured until this exists.
2. 2Add a reranker. Biggest quality jump for the least work.
3. 3Move to hybrid BM25 + dense retrieval.
4. 4Fix chunking on structure, with heading context prepended.
5. 5Add a refusal threshold and citation verification.

None of that involves changing the language model. In nearly every system I have worked on, the retrieval layer was the bottleneck and the model was the part that already worked.