positional encoding

Attention is permutation-invariant, so without position information a transformer cannot tell word order. The signal is knowing why, and why the field moved from absolute sinusoidal encodings to relative ones like RoPE and ALiBi that extrapolate to longer contexts.

Context windows are not arbitrary numbers; they fall out of attention's quadratic cost, KV-cache memory, and how the model was trained. The signal is naming all three causes and matching each to the technique that extends it.

Modern LLMs dropped learned absolute positions for RoPE and ALiBi to handle long context. The signal is knowing both encode RELATIVE position, and exactly how that enables length extrapolation past the trained window.

A model trained at 4K tokens can be stretched to 128K without full retraining. The signal is knowing why naive extrapolation fails and how PI, NTK-aware scaling, and YaRN each rescale RoPE frequencies differently.

RoPE is why modern LLMs extrapolate to longer contexts, and it's a rotation, not an addition. Coding it shows you actually understand how position enters attention. Here is the implementation.