Hugging Face Launches Open LLM Leaderboard v3

Hugging Face has released the third version of its influential Open LLM Leaderboard, introducing new evaluation benchmarks specifically designed to resist data contamination — a growing problem that has undermined the reliability of traditional AI benchmarks.

The new leaderboard replaces several legacy benchmarks with novel evaluation sets that are generated fresh each month and have never appeared in any training dataset. The benchmarks cover reasoning, coding, mathematics, instruction following, and multilingual capabilities, with questions designed to test genuine understanding rather than memorized patterns.

A key addition is the "Real-World Tasks" evaluation, which measures model performance on practical tasks like email drafting, data analysis, creative writing, and technical documentation. These evaluations use human preference ratings from a panel of over 1,000 diverse evaluators, providing a more holistic assessment than automated metrics alone.

The launch has already revealed some surprising results. Several models that scored highest on traditional benchmarks perform significantly worse on the contamination-resistant evaluations, suggesting their benchmark scores were inflated by exposure to test data during training. Conversely, some models with modest traditional scores perform much better on real-world tasks.

Hugging Face is making the evaluation framework open-source, allowing researchers and companies to run the same evaluations on their own models. CEO Clem Delangue noted that "reliable evaluation is one of the most important unsolved problems in AI" and that the new leaderboard represents "a step toward evaluations we can actually trust."