What is RAG, retrieval-augmented generation?

How RAG works

The term comes from a 2020 paper by Patrick Lewis and colleagues, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which described models combining pre-trained parametric and non-parametric memory for language generation — knowledge stored in the model's weights, plus knowledge fetched from an external index at query time. Today the term covers any system that retrieves first and generates second.

A typical pipeline has two halves. Indexing happens ahead of time: documents are split into chunks, each chunk is converted into an embedding vector, and the vectors go into a searchable index. Answering happens per question: the question is embedded, the nearest chunks are retrieved, and those chunks are placed in the prompt alongside the question, with an instruction to answer from them and cite them. Many systems combine vector search with keyword search, and some add a reranking step that reorders the retrieved passages before the best few reach the prompt.

What a RAG query costs

Most of the recurring cost is input tokens. Each query sends a system prompt, the retrieved passages and the question; the passages are usually the largest part. Retrieve five 800-token chunks and every query carries 4,000 tokens of context before the model writes a word. The answer itself is output tokens at the higher rate.

Indexing is cheaper but not free. Embedding models are priced per input token and at a fraction of a chat model's rate — our embeddings entry shows current rates — and the corpus is embedded once, then again only for documents that change. A vector database or search service adds its own hosting cost, outside the per-token bill.

Two levers cut the per-query bill. Retrieve fewer, better passages: precision in retrieval is a direct saving on every query. And keep the stable part — instructions, format rules — at the front of the prompt so prompt caching can bill it at the cached rate; retrieved passages differ per question and rarely cache.

RAG vs a long context window

Million-token context windows make it tempting to skip retrieval and send everything. For a corpus that fits, that is simpler — but every query then pays for the whole corpus as input. A thousand questions over a 500,000-token document set is half a billion input tokens; with retrieval, the same questions might send a few thousand tokens each. Caching narrows the gap when the corpus is identical on every call, but cached tokens still cost something, and Anthropic warns that recall degrades as context grows.

Some vendors also add a long-context surcharge past a threshold, which applies to the whole request. Retrieval keeps prompts under it. The practical rule: send everything when the material is small or every answer needs most of it; retrieve when each question needs a small part of a large or changing corpus.

RAG vs fine-tuning

RAG and fine-tuning solve different problems. RAG supplies facts at query time, so updating knowledge means updating the index, and answers can cite their sources. Fine-tuning changes the model's behaviour — format, tone, a narrow skill — and bakes knowledge in only loosely. In 2026 the choice is also shaped by availability: OpenAI is winding down its fine-tuning platform and the Gemini API offers no tuning, while RAG works with any model.

Keeping a RAG system cheap

Every lever comes back to the same fact: retrieved passages are input on every query. In rough order of payoff:

• Tune how much you retrieve. Test the number of passages and their size against answer quality; each extra passage is paid for on every question, forever.
• Order the prompt for caching. Fixed instructions and format rules first, so they cache; retrieved passages and the question after, since they change per query.
• Embed in bulk, once. OpenAI's Batch API accepts the embeddings endpoint at the batch discount, which suits indexing a large corpus; afterwards, re-embed only documents that changed.
• Store shorter vectors where accuracy allows. OpenAI's dimensions parameter shortens embeddings without losing their concept-representing properties, which lowers storage and search costs in the vector database.
• Match the model to the question. With good passages in front of it, a small model often answers well; route only the hard questions to a large one.

Measure the result as cost per answered question, including retrieval, rather than as a per-token rate.

Where RAG goes wrong

• Retrieving too much. Ten loosely relevant chunks cost more than three good ones and can dilute the answer.
• Bad chunking. Splitting mid-table or mid-argument retrieves fragments the model cannot use.
• Stale index. Documents that change without re-embedding return outdated passages with full confidence.
• No citations. Without source references in the answer, nobody can check what the model drew on.