The RAG Pipeline

TL;DR: RAG grounds a model in external knowledge: instead of relying on what the model memorized, you retrieve the most relevant chunks from a knowledge base at query time and place them in the prompt, so the model answers from real, current sources. It is the standard fix for hallucination and stale knowledge, and it updates by changing the index, no retraining. The pipeline: ingest and chunk documents, embed and index them, retrieve (and usually rerank) the top chunks for a query, then generate an answer grounded in them with citations.

Why RAG exists

A model's knowledge is frozen at training time and lives fuzzily in its weights, so it cannot cite sources, goes stale, and hallucinates when it lacks a fact. RAG decouples knowledge from the model: keep the knowledge in an external store and retrieve what is relevant per query. This makes answers grounded (traceable to sources), current (update the store, not the model), and scoped (only the right chunks enter the prompt, which is cheaper and more accurate than a giant context, see context window).

The two phases

Indexing (offline):

1. Ingest documents and parse them (handling PDFs, tables, layout).
2. Chunk them into passages (see chunking), the granularity that balances retrieval precision and context.
3. Embed each chunk with an embedding model and store the vectors in a vector index.

Querying (online): 4. Embed the query and retrieve the nearest chunks (often combined with keyword search, see hybrid search). 5. Rerank the candidates with a more precise model and keep the top few (see reranking). 6. Generate the answer from those chunks, with citations so it is verifiable.

Quality is bounded by retrieval

A crucial fact: RAG answers are only as good as what you retrieve. If the relevant chunk is not retrieved, no amount of generation quality recovers it, so retrieval recall is the ceiling for the whole system (see retrieval evaluation). Most RAG failures are retrieval failures: bad chunking, a domain-mismatched embedding model, or missing reranking. Evaluate retrieval and generation separately so you know which to fix.

Why interviewers probe this

RAG is the most common production LLM architecture, so designing one is the modal applied-AI design round. A strong answer walks the pipeline (chunk, embed, index, retrieve, rerank, generate with citations), states that it grounds the model to fix hallucination and staleness, and stresses that retrieval recall is the ceiling so most failures are retrieval failures. That framing, plus evaluating retrieval and generation separately, signals you can build and debug a real RAG system.

Common misconceptions

• "RAG fine-tunes the model on documents." It retrieves documents at inference; the model is unchanged and the knowledge is external.
• "RAG eliminates hallucination." It greatly reduces it but the model can still misuse context; you need citations and faithfulness checks.
• "Better generation fixes a bad RAG." If retrieval misses the chunk, generation cannot recover; retrieval recall is the ceiling.
• "RAG and long context are the same." RAG retrieves the few relevant chunks; long context stuffs everything (often worse and pricier).

Key takeaways

• RAG grounds the model by retrieving relevant chunks at query time and generating from them with citations.
• It fixes hallucination and stale knowledge and updates by changing the index, no retraining.
• The pipeline is chunk, embed, index, retrieve, rerank, generate.
• Retrieval recall is the ceiling, so evaluate retrieval and generation separately; most failures are retrieval failures.