preference optimization
Applied AI interview questions tagged preference optimization, across every topic.
3 questions · 1 unlocked for you
Concepts behind "preference optimization"
The curriculum that explains the ideas these questions test.
Core
RLHF: Reinforcement Learning from Human FeedbackRLHF is how a raw next-token predictor becomes a helpful, harmless assistant. It has three stages: supervised fine-tuning on demonstrations, training a reward model on human preference comparisons, then optimizing the model against that reward (with a KL penalty to stay close to the base). It aligns the model to human preferences that are hard to specify as a loss. Applied-AI interviews probe it because it explains why instruct models behave well, where alignment data comes from, and the failure modes (reward hacking, sycophancy).🧠 Foundations of LLMs & GenAISign in
Core
Reward ModelsA reward model turns human preference comparisons into a scalar score for any response, the signal RLHF optimizes against. It is trained on pairs of responses labeled by which a human preferred, learning to rank rather than to produce text. Its imperfections drive RLHF's failure modes: reward hacking (gaming the proxy) and staleness as the policy drifts off-distribution. Applied-AI interviews probe it because it explains where the alignment signal comes from and why it is gameable, and it generalizes to LLM-as-judge evaluation.🧠 Foundations of LLMs & GenAISign in
Core
DPO and Preference-Optimization VariantsDirect Preference Optimization aligns a model directly on preference pairs with a simple classification-style loss, skipping RLHF's separate reward model and RL loop, which makes alignment far simpler and more stable. A family of variants then relaxes DPO's requirements: SimPO removes the reference model, KTO removes the need for paired data, and ORPO merges SFT and alignment into one step. Applied-AI interviews probe it because DPO is now the common way teams align open models, and the variants show you understand what each requirement buys.🧠 Foundations of LLMs & GenAISign in