\n \n \n \n \n
← Signals
Recentmonthscience

An audit finds the most-cited preference leaderboard is optimised rather than merely measured

Singh and colleagues analysed roughly two million pairwise battles across 243 models from 42 providers over sixteen months of a public human-preference leaderboard. They reported undisclosed private testing under which a provider could evaluate many variants and publish only the best — one provider tested 27 private variants before releasing one publicly — that up to 26.5% of prompts were duplicates or near-duplicates, and that fine-tuning on arena-style prompts raised an arena win rate by 112% with no gain, or a slight loss, on general benchmarks. The last result is the important one: it separates performing well on the instrument from performing well on the thing the instrument was meant to stand for.

Observed — measured or witnessed, with the observation cited.

Subjects

Evidence

Verified — Every field is supported by a cited authoritative source.

Recorded on The difficulty of evaluating machine learning systems

  1. GLUE is saturated and SuperGLUE is built to replace it
  2. A human-parity claim in translation fails under document-level evaluation
  3. An audit finds the most-cited preference leaderboard is optimised rather than merely measured
  4. A systematic review of 445 benchmarks reports pervasive validity failures
  5. A randomised trial finds experienced developers slower with AI assistance
  6. The Commission proposes deferring the high-risk obligations
  7. The AI Index records both fast benchmark movement and doubts about benchmarks

Related

Related because they share a subject in the Atlas — never because the text looks similar.