kl divergence

These four quantities underlie cross-entropy loss, decision-tree splits, distillation, and the KL penalty in RLHF. The signal is deriving them from one another and pointing to exactly where each shows up in a real training loop.

Four drift tests, four different assumptions. The weak answer lists them; the strong one knows which handles high-dimensional embeddings, which needs binning, and which gives a calibrated p-value. Here is how to choose.