TL;DR: Tokenizers split text into subword units so a fixed vocabulary (typically 30k-100k+ tokens) can represent any string, including unseen words, with no out-of-vocabulary problem. BPE merges frequent character pairs bottom-up; WordPiece merges by likelihood; SentencePiece operates on raw text (spaces included) so it is language-agnostic. The consequences are concrete: you pay and wait per token, so token efficiency is cost and latency, and poor tokenization is why models struggle with digits, code, and underrepresented languages.
How to approach it. Define the problem tokenization solves (open vocabulary without OOV), contrast the three algorithms briefly, then spend most of the answer on why an applied engineer cares: cost, latency, context budget, and the failure modes. Interviewers reward the practitioner's lens, not a textbook recitation.
A strong answer. The problem. You cannot have a vocabulary entry for every possible word: the set is unbounded and rare words would never get enough training signal. Character-level avoids OOV but makes sequences very long and slow. Subword tokenization is the middle ground: a fixed vocabulary of frequent pieces, so common words are one token and rare or novel words decompose into known pieces. No OOV failures, manageable sequence length.
The algorithms:
- BPE (Byte Pair Encoding): start from characters and greedily merge the most frequent adjacent pair, repeatedly, until you hit the target vocab size. Byte-level BPE (GPT models) operates on raw bytes, so it handles any Unicode, emoji, and arbitrary text with no unknown token.
- WordPiece (BERT): similar bottom-up merging, but it merges the pair that most increases the training-corpus likelihood rather than raw frequency.
- SentencePiece (T5, LLaMA, many multilingual models): treats input as a raw stream including whitespace (encoding spaces as a special marker), so it needs no language-specific pre-tokenization. That makes it the natural choice for languages without spaces (Chinese, Japanese) and for multilingual models.
Why it matters in practice:
- Cost and latency are per token. API pricing and generation time scale with token count, so a verbose tokenization directly raises your bill and your latency. The same content costs different amounts depending on how efficiently it tokenizes.
- Context window is measured in tokens. What you can fit in the prompt and KV cache is a token budget, not a character budget.
- Multilingual penalty. A tokenizer trained mostly on English shatters other languages into many more tokens per word, so non-English users pay more, wait longer, and effectively get a smaller context window. A real fairness and cost issue, not a footnote.
- Numbers and code. Tokenizers split digit strings and code in inconsistent ways, so the model never sees clean digits, which is part of why models historically fumble arithmetic and precise formatting. Newer tokenizers special-case digits to mitigate this.
The point an interviewer wants: tokenization is not neutral plumbing. It sets your cost, your effective context, and several model weaknesses.
| Algorithm | Merge rule | Operates on | Used by |
|---|---|---|---|
| BPE | Most frequent adjacent pair | Chars or raw bytes | GPT family |
| WordPiece | Pair that maximizes corpus likelihood | Pre-tokenized words | BERT |
| SentencePiece | BPE/unigram over raw stream | Raw text incl. spaces | T5, LLaMA, multilingual |
Key takeaways
- Subword units buy an open vocabulary with no OOV at a sequence length you can afford.
- BPE merges by frequency, WordPiece by likelihood, SentencePiece by working on raw text including whitespace.
- Cost, latency, and context limits are all denominated in tokens, not characters.
- The English-centric tokenizer tax means non-English text inflates token counts and quietly shrinks the usable context.
What interviewers probe next.
- "Why not character- or word-level?" Character-level makes sequences too long and slow; word-level has an unbounded vocabulary and an OOV problem. Subword balances both.
- "Why do LLMs struggle with arithmetic and counting letters?" Numbers and characters get grouped into subword tokens inconsistently, so the model never sees clean digits or individual characters. Tokenization, not just reasoning, is part of the cause.
- "How does this affect non-English cost?" More tokens per word means higher cost, higher latency, and a smaller effective context for those languages.
- "What is a token roughly in English?" About 3-4 characters or ~0.75 words on average, but it varies widely by text.
Common mistakes.
- Describing tokenization as a preprocessing detail with no downstream consequences.
- Confusing the algorithms: BPE merges by frequency, WordPiece by likelihood, SentencePiece works on raw text including spaces.
- Forgetting that cost, latency, and context limits are all measured in tokens.
- Missing the multilingual token-inflation penalty.