Attention: how a model decides which words matter to which
You do not need the maths of attention to build with models, but you do need what it buys and what it costs. This lesson gives you the intuition and the one consequence that shapes every design decision later: cost grows with the square of the input.
TL;DR: Attention lets every token look at every other token and decide which ones matter for interpreting it. That is what solved context in language. It also costs work proportional to the square of the input length, and that single fact explains context-window limits, why long prompts get expensive fast, and half the architecture decisions in this course.
Where you are. You know text becomes tokens, and that tokens can be positioned so that meaning becomes distance. Attention is the mechanism that lets those pieces interpret each other. This is the most theoretical lesson in the module and the shortest path through it: enough to reason about cost and limits, nothing more. You will not be asked to do any maths.
The problem it solves
Take the sentence: the trophy did not fit in the suitcase because it was too big.
What does "it" refer to? Change one word, to too small, and the answer flips from the trophy to the suitcase. Nothing about the word "it" changed. The meaning came from its relationship to distant words.
Earlier language systems processed text in order, carrying a running summary forward. That works for short spans and degrades over long ones, because everything has to squeeze through a fixed-size memory that gets overwritten as you go. Information from thirty words back arrives faded, if it arrives at all.
What attention does instead
Attention drops the running summary. Instead, every token gets to look at every other token at once and decide how much each one matters to it.
For "it", the model computes a relevance score against every other token in the input, including "trophy", "suitcase", and "big". Those scores become weights, and the representation of "it" becomes a blend of the tokens it weighted highly. If "big" is in the sentence, "trophy" gets the weight. If "small" is, "suitcase" does.
Two properties follow, and both matter to you.
No distance penalty. A token thirty positions away is scored the same way as the one next door. Position is supplied separately rather than by how long information has survived. This is why models handle long-range reference well.
It is done for every token, against every token. Not just for the ambiguous ones. Every token in your input is scored against every other one.
The consequence you will live with
That second property is the one to carry out of this lesson.
If your input has 1,000 tokens, the model computes roughly 1,000 x 1,000 = one million pairwise scores. Double the input to 2,000 tokens and it is four million, not two. Work grows with the square of the length, not in proportion to it.
Sit with what that means in practice. Doubling your prompt does not double the cost of processing it, it roughly quadruples it. Ten times the prompt is about a hundred times the work.
This one fact explains a surprising amount:
- Why context windows have a limit at all, rather than being unbounded.
- Why "just put everything in the prompt" stops being viable, and why module 4 exists.
- Why long-context pricing and latency behave the way they do, rising faster than you expect from the token count alone.
- Why so much engineering effort goes into caching and reusing computed attention rather than recomputing it every turn.
There are techniques that soften the curve, and you will meet them later. None of them make it go away.
What you can safely not know
To be explicit, because beginner material tends to bury the useful part under the derivation: you do not need to know how the scores are computed, what the three projections are called, or why there is a square root in the formula. That is worth learning eventually and it is not load-bearing for anything you will build in this course.
What is load-bearing: every token sees every token, cost is quadratic in length, and position is added rather than implied.
"Eventually" is allowed to be today, though, and if you would rather learn it properly than take our word for what is safe to skip, the foundations are a lecture away.
Do this before moving on
Take a prompt you would realistically send, and write down its token count. Now write the counts for the same prompt at two, five, and ten times the length, alongside the relative attention work for each: 1x, 4x, 25x, 100x.
Then answer one question with those numbers in front of you: at what input size does "just include more context" stop being the cheap option for your use case? You have now done, roughly, the reasoning that decides whether a system needs retrieval. Module 4 is the detailed version.
Go deeper
- Attention mechanism is the mechanism this lesson deliberately skipped: queries, keys, values, and where the quadratic cost comes from precisely. Read it when you want the how.
- Transformer architecture puts attention in its surroundings and explains what the rest of the block does.
- Context window is the direct consequence of the cost curve above, and the prerequisite for the last lesson in this module.
- Positional encodings covers the "position is supplied separately" point properly. Optional now, useful later.
- Practice question: Why is the attention score divided by the square root of the key dimension? is the standard depth probe on this topic, and it is asked often enough to be worth owning.
- Practice question: How do you handle inputs longer than the context window? is where this lesson becomes a design decision.
Key takeaways
- Attention lets every token weigh every other token directly, with no penalty for distance, which is what fixed long-range reference in language.
- Cost grows with the square of input length. Double the input, roughly quadruple the work.
- That single curve explains context limits, long-prompt pricing, why retrieval exists, and why caching attention matters.
- You do not need the maths to build well. You do need the cost curve.
Check yourself before an interviewer does. Answer from memory first.
Your prompt grows from 2,000 to 8,000 tokens. Roughly how much more attention work is that?