RAG vs CAG: when to retrieve and when to cache

There are two ways to give a language model knowledge it was never trained on. You can fetch the relevant pieces at the moment of the question, which is RAG, retrieval-augmented generation. Or you can load everything up front and keep it warm, which is CAG, cache-augmented generation.

For a few years there was only one answer, because context windows were small and expensive. Everyone reached for RAG, including me. Context windows are now measured in millions of tokens and caching has made re-reading them cheap, so the default deserves another look. For a surprising number of systems, the whole retrieval layer is machinery you no longer need.

This is the comparison I wish I had been handed. It starts from scratch, then goes deep enough to actually make the decision.

The beginner version: a librarian and a colleague

RAG is a librarian. You ask a question. They hurry off into the stacks, find the three pages that look relevant, and hand them to you. You read those pages and answer from them. The librarian never gets tired and the library can be enormous, but everything depends on them picking the right pages. If they bring the wrong ones, you answer confidently from the wrong pages and never know.

CAG is a colleague who read the entire manual last week and still has it fresh. You just ask. There is no trip to the shelves, no chance of the wrong pages, no waiting. But they can only hold so much in their head, and when the manual is revised they have to sit down and read it all again.

Every real trade-off between the two falls out of that picture. RAG scales to a building full of books and can get the wrong ones. CAG cannot get the wrong ones because it has them all, and is capped by how much fits in one head.

What RAG actually does

RAG has two halves. One runs ahead of time, one runs on every question.

Ahead of time you ingest your documents, split them into chunks, turn each chunk into an embedding, which is a list of numbers describing its meaning, and store those in a vector database. On each question you embed the question the same way, search for the nearest chunks, usually rerank them, paste the best few into the prompt, and ask the model to answer from them.

1. 1Chunk the documents into passages of a few hundred tokens.
2. 2Embed each chunk and store it in a vector index.
3. 3Search at query time, ideally keyword and vector search together.
4. 4Rerank the candidates with a model that reads query and chunk as a pair.
5. 5Generate from the handful of chunks that survived.

The strength is that step 2 is the only step that grows with your corpus. Ten documents or ten million, the model still sees about five chunks. The weakness is that everything rests on steps 3 and 4. If the right chunk is not retrieved, the model does not know it is missing and writes something plausible instead.

What CAG actually does

CAG deletes steps 1 through 4. You put the entire corpus into the prompt, once, and let the model attend over all of it directly.

On its own that would be ruinous, because you would pay to process the whole corpus on every single question. The thing that makes it practical is the KV cache. When a model reads tokens it builds an internal representation of them, the keys and values its attention layers work over. That state is deterministic for a given prefix, so it can be computed once and reused. Cache it, and each new question only pays for the question.

1. 1Concatenate the corpus into one stable prefix.
2. 2Warm the cache with a single pass over it.
3. 3Ask questions against the warm prefix, paying only for the question and the answer.
4. 4Rebuild when the content changes or the cache expires.

On a hosted API this is exactly what prompt caching gives you, and CAG is really just prompt caching used deliberately as an architecture rather than as an optimisation. If you serve your own model you can go further and persist the KV cache to disk, then reload it per session.

The comparison that decides it

That last row is the one teams underestimate. A RAG pipeline is five components that each need tuning, monitoring and their own evaluation. CAG is a long string and an expiry time. When both approaches would work, the simpler one is not merely easier to build, it is the one that still works in eighteen months.

The cost maths, which usually settles the argument

Cache pricing follows the same shape across providers. Writing to the cache costs a little more than normal input, around 1.25 times. Reading from it costs around a tenth of normal input. That one-tenth figure is the whole ball game.

Because a cached token costs about a tenth of a fresh one, caching a corpus of 56,000 tokens costs roughly the same per query as retrieving and sending 5,600 fresh ones. That is your crossover. Under about 50,000 tokens, CAG is cheaper per query than RAG as well as simpler. Above it, RAG pulls ahead, and the gap grows in a straight line.

INPUT = 5.00                    # $ per 1M input tokens
CACHE_WRITE = INPUT * 1.25      # writing the prefix costs a little more
CACHE_READ = INPUT * 0.10       # reading it back costs about a tenth

def cag_per_query(corpus_tokens, queries_per_cache_window):
    """Every query re-reads the whole corpus, cheaply. The write amortises."""
    read = corpus_tokens * CACHE_READ / 1e6
    write = corpus_tokens * CACHE_WRITE / 1e6
    return read + write / max(queries_per_cache_window, 1)

def rag_per_query(chunk_tokens=700, k=8):
    """You only pay for what you retrieved, plus infrastructure."""
    return chunk_tokens * k * INPUT / 1e6

cag_per_query(200_000, queries_per_cache_window=50)   # $0.125
cag_per_query(30_000, queries_per_cache_window=50)    # $0.019
rag_per_query()                                       # $0.028

Note the second variable. Caches expire, typically after about five minutes by default, with a longer window available at a higher write price. If your traffic is steady, the write cost is amortised over hundreds of queries and disappears. If ten questions arrive per day at random intervals, you pay the full write cost almost every time, and CAG becomes the most expensive option on the table.

So the honest version of the cost question is not "how big is my corpus". It is "how big is my corpus, and how many questions arrive inside one cache window".

Where CAG wins, with real examples

The pattern is always the same: a bounded body of knowledge, many questions against it, content that does not change hourly.

• Product documentation support. A manual of 150 pages is roughly 90,000 tokens. Cache it and the assistant has read every page, including the cross-reference on page 12 that a chunk-based retriever would never connect to the question on page 130.
• Policy and handbook assistants. HR policies, expense rules, the employee handbook. Small, stable, and the same two hundred questions asked forever. This is the ideal CAG workload and it is very commonly over-engineered into RAG.
• One document, many questions. A contract review, an insurance policy wording, an annual report, a tender document. Load the document for the session and ask forty questions about it. Retrieval adds nothing here except a way to miss a clause.
• A stable prompt scaffold. Your system prompt, tool definitions, database schema, style guide and few-shot examples. Cache these even inside a RAG system. It is the cheapest win available and most teams leave it on the table.
• A single service or module of a codebase. Small enough to hold whole, and questions about code are exactly the questions that span files.

Where RAG wins, with real examples

• Company-wide knowledge. Wiki pages, tickets, past proposals, email threads. This is tens of millions of tokens and growing weekly. No context window solves it and none will.
• Product catalogues. Hundreds of thousands of items with prices and stock levels that change through the day. The data is too large and far too volatile to cache.
• Anything multi-tenant. Covered below. It is the strongest argument of the lot.
• Regulated work needing provenance. When an answer has to point at the exact document and clause it came from, retrieval hands you that identifier as a by-product. With CAG you have to ask the model to quote, then verify the quote is really in the source.
• Fast-moving content. News, prices, availability, anything where an answer five minutes stale is wrong. Re-indexing one document is cheap. Rebuilding a cache of a million tokens is not.

The permission problem nobody mentions

This is the argument that ends the debate for most business software, and it rarely comes up in the comparisons.

With RAG, access control is a filter. Every chunk carries metadata, the query adds WHERE tenant_id = ? or a role check, and a user simply cannot retrieve what they are not allowed to see. It costs nothing and it composes with everything else.

With CAG, the cache is the corpus. A cached prefix containing documents this user may not see is a leak waiting for the right question. The only fix is one cache per permission set. Five fixed roles, fine, that is five caches. Per-user or per-customer document access, and you have a cache per user, each paying its own write cost, each expiring while that user is at lunch. The economics collapse.

What good systems actually do: both

Treating this as a choice between two camps is the mistake. In production the interesting designs use both, because the two techniques fail in different places.

Cache the core, retrieve the tail. Some of your knowledge is small, stable and needed for every answer: the schema, the tone of voice, the top thirty policies. Cache that permanently. Retrieve only the long tail that does not fit. Most queries are then answered from the cached core and never touch the vector database at all.

Retrieve documents, then cache them. This is my favourite pattern and it dissolves the worst problem in RAG. Rather than retrieving chunks, retrieve whole documents, load the two or three relevant ones into a cached prefix, and answer from there. Retrieval only has to decide which documents are relevant, which is a far easier job than deciding which paragraph is, and once inside, the model sees the whole thing. It also solves the permission problem, because the retrieval step applies the filter before anything is cached.

Let the corpus grow into RAG. Start with CAG while the corpus is small, because it takes an afternoon. Measure. When it outgrows the window or the traffic pattern turns the cache economics against you, add retrieval in front of the cache you already have. Building the pipeline on day one for a corpus of 40,000 tokens is a cost with no benefit attached.

How to choose in five minutes

1. 1Measure the corpus in tokens, not megabytes or document counts. Everything follows from this number and most teams have never calculated it.
2. 2Does it fit in the context window with real headroom? If not, RAG, and the decision is over.
3. 3Must different users see different subsets? If yes, RAG, or retrieval deciding what gets cached, unless you have a handful of fixed roles.
4. 4Does the content change faster than your cache window? If yes, RAG.
5. 5Work out cost per query both ways at your actual traffic. Include how many questions arrive inside one cache window, because that is what amortises the write.
6. 6If CAG survives all five, use it. You have just avoided building and maintaining a retrieval pipeline, and no query will ever fail because the wrong paragraph was fetched.

One caveat on step 2. Fitting is not the same as working well. Recall across a very long context is good but not uniform, and a model asked to combine facts scattered across 800,000 tokens does measurably worse than one given the same facts in 50,000. Treat the context window as a budget to spend carefully, not a bucket to fill.

RAG is how you cope with a corpus too big to hold. If yours is not too big to hold, you are solving a problem you do not have.

That is the shift worth internalising. Retrieval used to be mandatory because context was scarce. It is now a deliberate choice you make when the corpus is genuinely large, genuinely volatile, or genuinely permissioned. When none of those apply, the best retrieval pipeline is the one you never had to build.