decoding

These are the knobs every LLM product tunes, and the interviewer wants to know what each does to the probability distribution and which setting fits which task. Including why factual and creative tasks want opposite settings.

How you decode from the model's probabilities shapes the output as much as the model. The signal is knowing the deterministic vs sampling methods and when each fits (factual vs creative). Here is the answer.

Token count is latency and dollars, not just style. 'Be concise' barely works, and max_tokens just truncates mid-sentence. Here is how to actually shape length without amputating answers.

Neural text degeneration is a known failure of likelihood-maximizing decoding, not a broken model. The fixes are specific decode-time knobs. Here is which lever to pull and what each one costs.

Generation has to stop somewhere, and the wrong stop rule either truncates answers or burns tokens on trailing garbage. Here is how the EOS token, stop strings, and max-token caps actually interact.

Beam search is the decoding algorithm everyone references and few can implement correctly. The signal is keeping k hypotheses by cumulative log-prob, summing logs (not multiplying probs), and handling completed sequences and length. Here is a correct implementation and the tradeoffs.

Implementing the KV cache proves you understand why decode is efficient: you append the new token's K/V and reuse the rest instead of recomputing. The signal is the append-and-attend-over-cache logic. Here is the implementation.

Every chat model's 'temperature' and 'top_p' knobs are a few lines of logit surgery. Implementing them shows you understand decoding, not just call it. Here is the from-scratch version.