arXiv:2605.30504v1 Announce Type: new Abstract: LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single
The proliferation of LLMs and their reliance on benchmarks makes the integrity of these evaluation systems a critical concern, as errors can propagate widely.
The discovery of systematic errors and mislabels in foundational LLM benchmarks fundamentally undermines current evaluation methods and calls into question reported model performance.
The focus on LLM evaluation will shift towards more robust, audited, and transparent benchmark creation and validation, moving beyond simple error propagation.
- · LLM auditing firms
- · Developers of new evaluation methodologies
- · Models that are genuinely robust beyond flawed benchmarks
- · LLMs with inflated performance due to benchmark 'gaming'
- · Research relying on unverified benchmark results
- · Current, un-audited LLM benchmark providers
Immediate re-evaluation of leading LLM performance claims and a push for more rigorous benchmark design.
Increased investment in, and adoption of, advanced techniques for benchmark validation, potentially leading to new industry standards.
A recalibration of what 'good' LLM performance truly means, fostering a more critical and nuanced understanding of AI capabilities beyond headline metrics.
This signal links to a primary source. Continuum Brief monitors and indexes it as part of the live intelligence stream — we do not republish source content.
Read at arXiv cs.CL