subword

Tokenization looks like plumbing until it explains your bill, your latency, and why the model fumbles numbers and rare languages. The signal is knowing how subword tokenizers work and the consequences that fall out of them.

BPE is the tokenizer behind GPT-style models, and implementing it proves you understand how subword vocabularies are built, not just that they exist. The signal is the iterative merge of the most frequent adjacent pair.

A from-scratch test of subword tokenization. The signal is greedy longest-match encoding against a fixed vocabulary, the continuation-prefix convention, and how WordPiece differs from BPE. Here is the implementation.