Design a production RAG system over 10M documents serving ~1,000 QPS at sub-second latency.

Question

Accepted Answer

TL;DR: Split it into an offline indexing path (chunk, embed, build hybrid BM25 + ANN indexes) and an online query path (retrieve top-100 hybrid, rerank to top-5 with a cross-encoder, generate with citations). Budget the latency, cache aggressively, and gate every change behind a retrieval eval set. The hard parts are chunking, hybrid fusion, and evaluation, not the LLM call. How to approach it. Clarify functional and non-functional requirements first: corpus churn rate, freshness SLA, p99 latency target, tenancy, and what "correct" means for this domain. Then separate the offline and online paths so the design stays legible and you can reason about each independently. A strong answer. Two pipelines, one built ahead of time and one on the hot path: Retrieval is hybrid: dense ANN (HNSW in a vector DB) catches semantic matches, BM25 catches exact terms, IDs, and rare tokens that embeddings blur. Fuse with Reciprocal Rank Fusion, then rerank the union with a cross-encoder, which is far more accurate than bi-encoder cosine because it attends jointly over query and passage. Reranking 100 candidates is the latency hotspot, so cap the candidate set and run it on a small batched GPU model. Generation streams with inline citations to the source chunks; if no chunk clears a relevance threshold, the system says it cannot answer rather than hallucinating. For 1,000 QPS: embedding and rerank models sit behind a batched inference server with autoscaling, the ANN index is sharded by tenant or topic, and a semantic cache on normalized queries absorbs the meaningful fraction of production traffic that is near-duplicate. 10M documents chunk to a few tens of millions of vectors; an HNSW index at that scale runs from tens of GB into the low hundreds, so shard it and scale read replicas rather than fighting disk. The part most candidates skip: evaluation. Build a labeled eval set and track retrieval metrics (recall@k, nDCG) separately from answer metrics (faithfulness or groundedness, answer relevance via an LLM judge plus spot human review). No chunking or embedding change ships without moving these numbers. Separating the two layers tells you whether a regression came from retrieval or generation. Key takeaways Two paths: an offline indexing pipeline and a latency-budgeted online path; design and scale them separately. Hybrid (ANN + BM25) then cross-encoder rerank is the workhorse; dense alone fails on codes, IDs, and names. The cross-encoder rerank is the latency hotspot; cap candidates at ~100 and batch it on GPU. Evaluate retrieval and answer quality as separate metrics; nothing ships without moving them. What interviewers probe next. "Chunk size?" Start 400-600 tokens with ~15% overlap and respect document structure; too small loses context, too large dilutes the embedding and wastes the context window. Tune against recall@k, do not guess. "Why rerank if ANN already ranks?" Bi-encoders embed query and doc independently; a cross-encoder sees them together and is much sharper on precision@5, which is what the generator actually consumes. "How do you handle stale or deleted docs?" Soft-delete with metadata filters at query time plus periodic index compaction; tie freshness to the CDC stream from the source of truth. "Multi-tenant isolation?" Partition indexes and enforce tenant filters server-side so one customer can never retrieve another's chunks. Common mistakes. Going straight to model serving and never mentioning evaluation, so there is no way to know if a change helped. Pure dense retrieval, then losing exact-match queries (error codes, names, SKUs) that BM25 would have caught. Treating recall@k as the goal; high recall with a weak reranker still feeds the generator noise. No "I don't know" path, so the system confidently answers from irrelevant context.