How do RoPE and ALiBi encode position, and why do they extrapolate better than learned positions?

Modern LLMs dropped learned absolute positions for RoPE and ALiBi to handle long context. The signal is knowing both encode RELATIVE position, and exactly how that enables length extrapolation past the trained window.

Updated Aug 2026 · Grounded in real Applied AI Engineer interview loops and written to a senior-engineer editorial bar.