language models

Perplexity is the classic LM metric. The signal is knowing it is just exponentiated cross-entropy, what it intuitively measures, and why a low perplexity does not mean a good product.

Before transformers, this was the language model, and it is still the cleanest way to show you understand what perplexity actually measures. Smoothing is the part people get wrong. Here is the implementation.