AppliedAIPrep logoAppliedAI/Prep
← LLM & GenAI Fundamentals / 03
hard★ EssentialOpenAIAnthropicCohere

Walk through RLHF, then explain DPO and why it has largely displaced PPO-based RLHF.

Alignment is now table stakes even outside 'safety' roles. The interviewer wants the three-stage RLHF pipeline and a crisp account of why DPO dropped the reward model entirely. Here is the answer with the mechanism, not the buzzwords.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.

TL;DR: RLHF is three stages: supervised fine-tuning, train a reward model on human preference comparisons, then optimize the policy against that reward with PPO and a KL penalty to the reference model. DPO collapses this: it shows the optimal RLHF policy has a closed form in terms of the preference data, so you can optimize a simple classification-style loss on preferred-vs-rejected pairs directly, with no separate reward model and no RL loop. It is more stable and cheaper, which is why it now dominates.

rendering diagram…

How to approach it. Give the RLHF pipeline first (most candidates can name stages but not say what each optimizes), then explain DPO's key insight (the reward model is implicit) and the practical reasons it won. Name the KL-to-reference term in both, because that is what keeps the model from degenerating.

A strong answer. RLHF, three stages:

  1. SFT: fine-tune the base model on high-quality demonstrations so it follows instructions at all.
  2. Reward model: collect human comparisons (given a prompt and two responses, which is better) and train a model to score responses, fitting the Bradley-Terry preference likelihood.
  3. RL optimization (PPO): treat the LLM as a policy and optimize it to maximize the reward model's score, with a KL-divergence penalty against the SFT/reference model so it does not drift into gibberish that games the reward (reward hacking). PPO here is finicky: you must sample from the model during training, keep a reward model, a value/critic model, and the reference model all in memory, and tune carefully.

DPO: Direct Preference Optimization observes that for the KL-constrained RLHF objective, the optimal policy is a known closed-form function of the reference policy and the reward. Invert that relationship and the reward cancels out, leaving a loss you can compute directly on preference pairs: increase the relative log-probability of the preferred response over the rejected one, scaled by a temperature beta, relative to the reference model. Per the DPO paper (Rafailov et al., 2023), it is "stable, performant, and computationally lightweight, eliminating the need for sampling from the LM during fine-tuning or performing significant hyperparameter tuning." No separate reward model, no online sampling, no RL loop, just a supervised-style objective on pairs.

Worth name-dropping the next step: GRPO (from DeepSeekMath, used in DeepSeek-R1) drops the value/critic model and estimates advantage by normalizing rewards within a group of sampled responses, cutting memory further. The throughline: the field keeps removing moving parts from RLHF.

Key takeaways

  • RLHF is SFT, then a reward model on preference comparisons, then PPO against that reward with a KL leash to the reference policy.
  • DPO's insight is that the KL-constrained optimum has a closed form, so the reward model becomes implicit and the loss reduces to classification on preferred-vs-rejected pairs.
  • The KL-to-reference term is non-negotiable in both: it is what prevents reward hacking and degeneration.
  • DPO wins on stability and cost (no reward model, no online sampling, no RL loop); GRPO pushes the trend further by dropping the critic.

What interviewers probe next.

  • "Why the KL penalty?" Without it, the policy overfits the reward proxy (reward hacking) and degenerates; KL keeps it close to a sane reference distribution.
  • "DPO downsides?" It optimizes a fixed offline preference dataset, so it cannot explore new responses the way online RL can, and it is sensitive to beta and to preference-data quality.
  • "RLHF vs RLAIF / Constitutional AI?" RLAIF replaces human labels with model-generated preferences guided by a set of principles, scaling the preference data.
  • "When would you still use PPO?" When you want online exploration or a reusable reward model across tasks.

Common mistakes.

  • Listing the three stages without saying what each optimizes.
  • Forgetting the KL-to-reference term, which is the whole reason the model does not collapse.
  • Claiming DPO "is just RLHF" rather than explaining it removes the reward model and RL loop via a closed-form result.
  • Inventing benchmark numbers; speak to the mechanism and cite the result qualitatively.
HOW DID IT GO?
0
UP NEXT ON YOUR JOURNEY
Next in this trackEncoder-only vs decoder-only vs encoder-decoder: when do you use each, and why are decoder-only models dominant?Next in this trackWhat causes LLM hallucinations, and how do you reduce them in a production feature?Next in this trackExplain tokenization (BPE, WordPiece, SentencePiece) and why it quietly drives cost, latency, and quality.
DISCUSSION · 0

No comments yet — be the first to share your approach.