TL;DR: LLMs hallucinate because they are trained to produce fluent, probable continuations, not to know what they do not know. There is no built-in truth check and no access to facts outside their weights. Reduce it by grounding answers in retrieved sources (RAG) with citations, giving the model an explicit "I don't know" path, constraining claims to the provided context, and measuring faithfulness so you know your rate. You contain and measure it; you do not eliminate it.
How to approach it. Explain the root cause (next-token objective, no truth signal, closed-book knowledge gaps) before jumping to fixes, because every fix follows from the cause. Then layer the mitigations and put weight on measurement, since "how do you know it improved" is the real test and most candidates skip it.
A strong answer. Why it happens. A model predicts the most probable next token given context. The objective optimizes fluency and plausibility, not factual accuracy, and the network has no internal flag for "I am uncertain" versus "I know this." Hallucination is worst when the answer needs facts not in the weights: recent events, private data, long-tail specifics. The model then confabulates something that reads right. This is how the model works, not a bug you patch away.
Mitigations, layered from highest leverage down:
- Grounding (RAG). Retrieve relevant sources and instruct the model to answer only from them, with citations the user can click. This is the single biggest lever for knowledge hallucinations: facts come from a controlled corpus, not the model's memory.
- Abstention. Give an explicit "if the context does not contain the answer, say you do not know" path plus a grounding threshold. A system that declines beats one that confidently invents.
- Constrain and verify. Restrict the model to the provided context, and for high-stakes outputs add a verification pass that checks every claim traces to a source.
- Prompting and decoding. Clear instructions, few-shot examples, and lower temperature cut creative drift on factual tasks. Necessary, not sufficient.
- Fine-tuning / RLHF improves calibration and reduces confident wrongness, but it does not inject missing facts. That is retrieval's job.
Measurement. You cannot improve what you do not measure. Track a faithfulness/groundedness rate on an eval set (does every claim trace to a source), scored by an LLM judge calibrated against human labels, plus production signals like citation clicks and user corrections. State your hallucination rate honestly. "Zero hallucination" in anyone's pitch is a red flag.
| Lever | Fixes | Cost | Limit |
|---|---|---|---|
| RAG grounding | Missing/stale facts | Retrieval infra, latency | Only as good as the corpus |
| Abstention | Confident wrong answers | A few declined queries | Tune the threshold or it over-refuses |
| Verification pass | Unsupported claims | Extra model call | Adds latency and spend |
| Lower temperature | Creative drift | Near zero | Does not close knowledge gaps |
Key takeaways
- Hallucination is the next-token objective working as designed, not a defect, so frame fixes as containment.
- RAG with clickable citations is the highest-leverage move for factual errors; abstention is the cheap safety net.
- Fine-tuning improves calibration but never injects fresh facts; that is retrieval's job.
- Ship a faithfulness metric on day one or quality regressions stay invisible.
What interviewers probe next.
- "RAG or fine-tuning to fix factual errors?" RAG. Facts change and fine-tuning bakes in stale facts while still hallucinating; retrieval grounds and updates without retraining.
- "How do you detect hallucination automatically?" Groundedness scoring of each claim against retrieved context, flagging unsupported claims. It is imperfect, so pair it with user-visible citations.
- "Does lower temperature solve it?" It reduces variance, not the knowledge gap. Necessary, not sufficient.
- "What about reasoning models or self-consistency?" Chain-of-thought and sampling multiple answers then voting catch some errors at a latency and cost hit; you still need grounding for facts.
Common mistakes.
- "Just prompt it better" as the whole answer, ignoring grounding and abstention.
- Fine-tuning to add facts, then fighting stale and still-confident errors.
- Claiming a method eliminates hallucination rather than reducing and measuring it.
- Shipping with no faithfulness metric, so degradation goes unnoticed until a customer complains.