alignment
Applied AI interview questions tagged alignment, across every topic.
13 questions · 1 unlocked for you
Concepts behind "alignment"
The curriculum that explains the ideas these questions test.
Core
RLHF: Reinforcement Learning from Human FeedbackRLHF is how a raw next-token predictor becomes a helpful, harmless assistant. It has three stages: supervised fine-tuning on demonstrations, training a reward model on human preference comparisons, then optimizing the model against that reward (with a KL penalty to stay close to the base). It aligns the model to human preferences that are hard to specify as a loss. Applied-AI interviews probe it because it explains why instruct models behave well, where alignment data comes from, and the failure modes (reward hacking, sycophancy).🧠 Foundations of LLMs & GenAISign in
Core
Reward ModelsA reward model turns human preference comparisons into a scalar score for any response, the signal RLHF optimizes against. It is trained on pairs of responses labeled by which a human preferred, learning to rank rather than to produce text. Its imperfections drive RLHF's failure modes: reward hacking (gaming the proxy) and staleness as the policy drifts off-distribution. Applied-AI interviews probe it because it explains where the alignment signal comes from and why it is gameable, and it generalizes to LLM-as-judge evaluation.🧠 Foundations of LLMs & GenAISign in
Core
Constitutional AI and RLAIFRLAIF (RL from AI Feedback) replaces human preference labels with AI-generated ones, scaling alignment past the human-labeling bottleneck. Constitutional AI is Anthropic's specific approach: the model critiques and revises its own outputs against a written set of principles (a constitution), generating the preference data from those principles. The win is scalability, consistency, and explicit, editable values; the risk is the AI judge's own biases. Applied-AI interviews probe it because it is how alignment scales and how values become explicit and auditable.🧠 Foundations of LLMs & GenAISign in
Core
DPO and Preference-Optimization VariantsDirect Preference Optimization aligns a model directly on preference pairs with a simple classification-style loss, skipping RLHF's separate reward model and RL loop, which makes alignment far simpler and more stable. A family of variants then relaxes DPO's requirements: SimPO removes the reference model, KTO removes the need for paired data, and ORPO merges SFT and alignment into one step. Applied-AI interviews probe it because DPO is now the common way teams align open models, and the variants show you understand what each requirement buys.🧠 Foundations of LLMs & GenAISign in
Advanced
Policy Optimization: PPO and GRPOPPO and GRPO are the reinforcement-learning algorithms that optimize an LLM against a reward, the RL step in RLHF and in training reasoning models. PPO is the established workhorse, updating the policy in small, clipped steps to stay stable; GRPO (used by DeepSeek-R1) drops PPO's separate value network and instead normalizes rewards within a group of samples, which is simpler and cheaper for LLMs. Applied-AI interviews probe it because it explains how alignment and reasoning training actually run, and why RL on verifiable rewards scales.🧠 Foundations of LLMs & GenAI🔒 Premium