Tokenization
Models do not read characters or words; they read tokens, subword chunks produced by an algorithm like BPE that maps text to integer IDs. Tokenization decides how many tokens a piece of text costs (driving price, latency, and context usage), why models miscount letters or fumble rare words, and why non-English text is more expensive. Applied-AI interviews probe it because token accounting is the first thing that bites a production LLM bill.
TL;DR: An LLM never sees raw text. A tokenizer splits text into subword units (tokens) and maps each to an integer ID; the model only ever processes those IDs. Subword schemes like Byte Pair Encoding keep common words as single tokens and break rare ones into pieces, so there is no unknown word, but token counts vary wildly by language and content. Because you pay, wait, and fill the context window per token, tokenization is the unit of account for every production LLM system.
Why models read tokens, not words or characters
Character-level models would make sequences impractically long; word-level models would need an enormous vocabulary and still choke on unseen words. Subword tokenization is the compromise: a fixed vocabulary (say 50k-200k entries) of frequent words and word fragments. Common words ("the", "model") are single tokens; rarer or novel words ("tokenization", "antidisestablishment") split into known pieces. The model maps each token to an integer ID, and every input is just a sequence of those IDs.
The dominant scheme is Byte Pair Encoding (BPE): start from characters and repeatedly merge the most frequent adjacent pair into a new token, building up a vocabulary of useful chunks. Modern tokenizers operate on bytes (byte-level BPE), so any Unicode, emoji, or code is representable, there is never a true "unknown token", worst case a string falls back to individual bytes.
Why token accounting runs your bill
Everything downstream is measured in tokens, so tokenization is the unit of account:
- Cost. APIs charge per input + output token. A prompt that tokenizes into 1,200 tokens costs twice one that tokenizes into 600.
- Latency. Generation is per-token, and the prompt must be processed (prefill) before the first token appears, so token count drives time-to-first-token and total latency.
- Context budget. The context window is measured in tokens, so verbose or token-inefficient text fills it faster (see the context window).
A subtle, costly fact: non-English text and non-Latin scripts tokenize into far more tokens per unit of meaning, because the tokenizer was trained mostly on English. The same sentence in Hindi or Chinese can cost several times more tokens than in English, a real fairness and cost issue for global products (see multilingual models).
Why models "can't count letters"
A model that fails "how many r's in strawberry?" is not dumb, it never saw the letters. It saw the tokens for "straw" and "berry" (or similar), so character-level questions are genuinely hard for it. The same goes for reversing strings or arithmetic on digits: the tokenization hides the structure the task needs. Knowing this explains a whole class of "silly" LLM failures.
Why interviewers probe this
Token accounting is the first thing that surprises teams shipping LLMs: a feature that worked in a demo blows the budget at scale because prompts are token-heavy. A strong answer connects tokenization to the three things that bite, cost, latency, and context budget, and can explain why a multilingual product costs more and why letter-counting fails. It signals you think in the units the system actually runs on.
Common misconceptions
- "Tokens are words." Tokens are subword chunks; one word can be several tokens, and a token can span a space or punctuation.
- "There are unknown words." Byte-level BPE always represents any input (worst case, byte by byte), so there is no out-of-vocabulary failure.
- "Token count equals word count." A rough rule is ~0.75 words per token for English, but it varies by content and breaks badly for other languages and code.
- "All languages cost the same." Non-English text usually costs more tokens for the same meaning, because the tokenizer is English-centric.
Key takeaways
- A tokenizer maps text to subword token IDs; the model only ever processes those IDs.
- BPE (byte-level) builds the vocabulary by merging frequent pairs, so there is no true unknown token.
- Tokens are the unit of cost, latency, and context budget, the first lever in any LLM cost model.
- Tokenization explains miscounting letters, fumbling rare words, and the higher cost of non-English text.
Check yourself before an interviewer does. Answer from memory first.
The same prompt costs three times more tokens in Hindi than in English. What is the first-order explanation?