Attention and Self-Attention

Attention turns each token into a query, key, and value, scores every query against every key, softmaxes those scores into weights, and returns the weighted sum of values, so each token pulls in information from the others. Self-attention does this within one sequence. The all-pairs scoring is why cost grows with the square of sequence length, which in turn explains context limits, long-prompt expense, and the KV cache. Applied-AI interviews probe it because it links architecture to cost and latency in one mental model.