← 🧠 Foundations of LLMs & GenAINEXT IN FOUNDATIONS OF LLMS & GENAITemperature and Sampling→
Core
Positional Encodings (RoPE and ALiBi)
Attention is order-blind, so models inject token position separately. Modern LLMs use relative schemes: RoPE rotates query/key vectors by an angle proportional to position so the attention score depends only on the offset between tokens, and ALiBi adds a distance penalty to attention scores. Both extrapolate to longer sequences far better than learned absolute positions, which is why RoPE-with-scaling is how context windows get extended. Applied-AI interviews probe it because it explains how long-context models are built.
a free account unlocks the core curriculum tier · no card
RELATED CONCEPTS
PRACTICE THIS IN REAL QUESTIONS
LLM & GenAI FundamentalsWhy do transformers need positional encoding, and how do sinusoidal, RoPE, and ALiBi differ?→LLM & GenAI FundamentalsHow do RoPE and ALiBi encode position, and why do they extrapolate better than learned positions?→LLM & GenAI FundamentalsWhy is an LLM's context window limited, and how do models extend it?→Coding & DSAImplement Rotary Position Embedding (RoPE) applied to query and key vectors.→LLM & GenAI FundamentalsHow do RoPE scaling methods (position interpolation, NTK, YaRN) extend a model's context window?→LLM & GenAI FundamentalsWhy do transformers scale attention scores by 1/√d_k, and what breaks if you skip it?→
COMPANIES THAT ASSUME THIS