TL;DR: Default to prompting (cheapest, fastest to iterate). Add RAG when the model lacks knowledge: private, fresh, or large-corpus facts it should ground answers in. Reach for fine-tuning when the model lacks a behavior: a consistent format, tone, or task skill that prompting cannot reliably produce. They compose, and the usual order is prompt, then RAG, then fine-tune only if needed.
How to approach it. Reframe the question as "what is actually missing": knowledge or behavior. Name your default and the two or three signals that move you off it (cost, latency, data freshness, volume of examples). Make the call, then state when you would combine them.
A strong answer. The decision turns on the failure you are solving.
- Prompting (instructions, few-shot examples, structured output) is the default: zero training cost, instant iteration, and a frontier model is already capable. Start here and only escalate when prompting demonstrably plateaus.
- RAG fixes a knowledge gap. Use it when answers must be grounded in private docs, data that changes often, or a corpus too large for the context window. RAG also gives you citations and lets you update knowledge by re-indexing instead of retraining, which is why it is the workhorse for enterprise Q&A. It does not teach the model a new skill or style.
- Fine-tuning fixes a behavior gap: a strict output format, a domain tone, a classification or extraction task where you have hundreds to thousands of labeled examples and prompting is inconsistent or too verbose. LoRA/QLoRA make this cheap (you train a small adapter, not the full model). Fine-tuning bakes in how to respond; it is a poor and expensive way to inject facts, which drift and would force a retrain.
The mapping interviewers want to hear is gap to tool, not three definitions:
| Symptom | Root gap | Reach for | Why not the others |
|---|---|---|---|
| Wrong/missing private facts | Knowledge | RAG | Fine-tuning bakes facts that go stale and hallucinate |
| Stale answers as data changes | Knowledge (freshness) | RAG (re-index) | Retraining per update is slow and expensive |
| Inconsistent format/tone at volume | Behavior | Fine-tune (LoRA) | Prompting drifts; RAG adds no style |
| Capable model, just needs steering | Neither yet | Prompting | Cheapest, instant iteration |
They compose: a fine-tuned model that follows your format, fed retrieved context via RAG, prompted with clear instructions, is a common production stack. The defensible position: prompt first, add retrieval when grounding fails, fine-tune last and only for behavior. Reverse it (fine-tune first) and you spend weeks and a training budget on something a better prompt would have solved in an afternoon.
Key takeaways
- Diagnose the gap before picking a tool: knowledge gaps go to RAG, behavior gaps go to fine-tuning, everything else stays in the prompt.
- Default to prompting and escalate only on a demonstrated plateau; the signals that move you are cost, latency, freshness, and example volume.
- Fine-tuning teaches how to respond, not what is true; injecting facts that way buys you hallucinations and a retrain treadmill.
- These compose into one stack (fine-tuned format + RAG context + clear prompt); the order is prompt, RAG, fine-tune, never the reverse.
What interviewers probe next.
- "The model gives outdated facts. RAG or fine-tune?" RAG. Facts change; retrieval updates without retraining, and fine-tuning facts invites hallucination and staleness.
- "You need a rigid JSON shape every time?" Prompt with a schema and constrained decoding first; fine-tune only if it still drifts at volume.
- "How much data for fine-tuning?" Often hundreds to a few thousand high-quality examples for a focused task with LoRA; quality and consistency matter more than raw count.
- "Cost/latency tradeoffs?" RAG adds retrieval latency and an index to operate; fine-tuning adds a training pipeline and model-versioning burden; prompting is cheapest to run and change.
Common mistakes.
- Defining all three instead of making a call with reversal conditions.
- Fine-tuning to add knowledge, then fighting hallucinations and stale facts.
- Jumping to fine-tuning before exhausting prompting and RAG, paying cost and iteration speed for no gain.
- Treating them as mutually exclusive rather than a composable stack.