83How do you detect and prevent benchmark contamination, and why are public LLM leaderboards often inflated?▼hardOpenAIGoogle DeepMindHugging Face1 replies◆ premiumIf a benchmark leaked into pretraining, the score measures memorization, not ability. The signal is the detection methods (n-gram overlap, canaries, perturbation tests) and why fresh held-out evals matter.Open full answer →
84How do you design an offline LLM eval harness so the numbers are reproducible and comparable?▼mediumHugging FaceOpenAIMicrosoft2 replies◆ premiumThe same model can score ten points apart on MMLU depending on prompt format and scoring method. The signal is knowing the knobs (log-prob vs generation, few-shot, normalization) that make evals reproducible.Open full answer →
