rope

Attention is permutation-invariant, so without position information a transformer cannot tell word order. The signal is knowing why, and why the field moved from absolute sinusoidal encodings to relative ones like RoPE and ALiBi that extrapolate to longer contexts.

Modern LLMs dropped learned absolute positions for RoPE and ALiBi to handle long context. The signal is knowing both encode RELATIVE position, and exactly how that enables length extrapolation past the trained window.

RoPE is why modern LLMs extrapolate to longer contexts, and it's a rotation, not an addition. Coding it shows you actually understand how position enters attention. Here is the implementation.