layernorm

LayerNorm sits in every transformer, and implementing it checks that you know what it normalizes over and why. The signal is normalizing across the feature dimension per token, the learnable scale/shift, and the RMSNorm simplification modern LLMs adopted.

BatchNorm is one of the most-asked deep-learning questions, and the trap is the train/inference difference. The signal is what it normalizes, why it stabilizes and speeds training, and why it switches to running statistics at inference. Here is that answer.