dpo

Alignment is now table stakes even outside 'safety' roles. The interviewer wants the three-stage RLHF pipeline and a crisp account of why DPO dropped the reward model entirely. Here is the answer with the mechanism, not the buzzwords.

DPO simplified RLHF, and a family of variants now trade off what data and reference model they need. The signal is knowing what each variant removes or changes (reference model, paired data, separate SFT stage). Here is the answer.

DPO is simple and stable, but it can push down the probability of chosen responses, overfit the preference margin, and amplify length bias. The signal is naming these and the variants that fix each.