reward modeling

Optimize any proxy hard enough and the model finds the exploit, not the goal. Sycophancy, padding, and fake citations are reward hacking. Here is why it happens and the controls that actually hold.

Push PPO hard enough and true quality peaks then falls while the reward keeps climbing. The signal is knowing the Gold-vs-proxy gap, the KL budget that bounds it, and how you actually measure when to stop.

Replace a learned reward model with a checker that returns right-or-wrong, and reward hacking mostly disappears. The signal is why verifiable rewards beat learned ones for math and code, and where they break.

A reward model is only as good as its labels, and human preference labels are noisy and inconsistent. The signal is measuring inter-annotator agreement and the concrete moves that lift label quality.

A reward model turns pairwise preferences into a scalar signal RLHF can optimize. The signal is the Bradley-Terry loss, the base-model and head choices, and how you validate it before trusting it.

RLAIF replaces expensive human labels with an LLM's preferences, which scales but inherits the labeler model's biases. The signal is how AI feedback is collected and where it quietly fails.