TL;DR: Self-attention is permutation-invariant (it treats the input as a set), so position must be injected explicitly or the model cannot distinguish "dog bites man" from "man bites dog." Original transformers added fixed sinusoidal or learned absolute position embeddings. Modern LLMs use relative schemes: RoPE rotates the query/key vectors by position so the attention score depends on relative distance (LLaMA, Mistral), and ALiBi adds a distance-based bias to attention scores (BLOOM, MPT). Relative methods extrapolate to longer sequences far better than absolute ones.
How to approach it. Start with why position is needed (attention is order-agnostic), then walk the evolution from absolute to relative encodings, emphasizing the property that drove the shift: extrapolation to longer contexts than seen in training. That property is what the interviewer is really probing.
A strong answer. Why it is needed. Self-attention computes a weighted sum over all tokens by content similarity, with no inherent notion of order. Permute the inputs and you permute the outputs identically. The model literally cannot tell word order unless you inject position, and word order carries meaning, so position information is not optional.
Absolute encodings (original). The first transformers added a position vector to each token embedding. Sinusoidal: fixed sine and cosine functions of position at different frequencies, so each position gets a unique pattern and the model can in principle infer relative offsets. Learned absolute: a trainable embedding per position (BERT, GPT-2). Both are absolute (they encode "position 5"), and learned ones cannot represent positions beyond the max length seen in training, so they do not extrapolate.
Relative encodings (modern). The field moved to encoding relative distance, which generalizes better:
- RoPE (Rotary Position Embedding). Rotate the query and key vectors by an angle proportional to their absolute position before the dot product. The attention score then depends on the difference of the rotations, so it effectively encodes relative position while being applied multiplicatively inside attention. Used in LLaMA, Mistral, and most current open models. It extrapolates reasonably and supports context-extension tricks (interpolating or scaling the rotation frequencies to stretch the usable window).
- ALiBi (Attention with Linear Biases). Add a bias to the attention scores that grows linearly with the distance between query and key (penalize attending to far-away tokens), with a per-head slope. No position embeddings at all, and it extrapolates to longer sequences than trained on by design. Used in BLOOM, MPT.
| Scheme | Absolute or relative | Mechanism | Extrapolation |
|---|---|---|---|
| Sinusoidal | Absolute | Added position vector (fixed) | Limited |
| Learned absolute | Absolute | Trained per-position embedding | None past trained length |
| RoPE | Relative | Rotate Q/K by position | Good, scalable via freq interpolation |
| ALiBi | Relative | Linear distance bias on scores | Strong, by design |
The throughline an interviewer rewards: position is mandatory because attention is order-agnostic, and the move from absolute to relative encodings (RoPE, ALiBi) was driven by wanting models that handle and extend to longer contexts than they were trained on.
Key takeaways
- Attention is permutation-invariant, so position must be injected or word order is invisible to the model.
- Absolute encodings tie meaning to a fixed index; learned ones hit a hard wall at the trained max length.
- RoPE rotates Q/K so scores depend on relative distance; ALiBi biases scores by distance with no embedding at all.
- Relative schemes extrapolate to longer contexts, and RoPE frequency scaling is the standard lever for extending a window.
What interviewers probe next.
- "Why do relative encodings extrapolate better?" They depend on token distance, not an absolute index, so they are not tied to specific positions seen in training. ALiBi is explicitly designed for length extrapolation, and RoPE supports frequency scaling to extend context.
- "How do you extend a model's context window?" For RoPE, interpolate or scale the rotary frequencies (position interpolation, NTK-aware scaling) and usually fine-tune on longer sequences.
- "Sinusoidal vs learned absolute?" Sinusoidal is fixed and can extrapolate somewhat; learned absolute is trainable but capped at the trained max length.
- "Connection to lost-in-the-middle?" Position handling interacts with how well a model uses information across a long context; it is part of why long-context quality is uneven.
Common mistakes.
- Not explaining why position is needed (attention's permutation invariance).
- Treating all positional schemes as equivalent; absolute vs relative is the key axis and drives extrapolation.
- Describing RoPE as an added embedding; it is a rotation applied to Q/K so scores depend on relative position.
- Forgetting that learned absolute encodings cannot exceed the trained sequence length.