Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards

• URL: http://arxiv.org/abs/2608.15980v1
• Date: Mon, 17 Aug 2026 00:19:52 GMT
• Title: Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards
• Abstract summary: We show that preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail.<n>On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all.<n>Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail. We measure what that detail buys. On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all, expert and crowd annotators assign a different majority label to 23.6% and name the opposite winner on 9.2%; on the 246 comparably unanimous MT-Bench cells, benchmark authors and recruited experts differ on 30.5% and reverse on 8.5%. Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced. That invariance is far weaker evidence than it looks, and we quantify how weak. Switching pools moves a model's win rate by 1.9pp (SD), one adjacent pair in our own leaderboard sits 0.8pp apart and had a 38% chance of swapping, and an item-level bootstrap displaces at least one model in 28% of resamples. The observed zero is the common outcome, not a property of aggregation: on the same measured perturbation, a ten-model leaderboard is displaced with probability 0.86 and a twenty-model leaderboard with probability 0.9997. Reporting a six-model leaderboard is safe; the safety does not generalise, and everything that consumes labels per item is not safe at any size. We make the distinction precise, show that a widely used dataset's stated assumption of no intra-group annotator variability is false, and show that an LLM judge tracks the crowd pool over the expert pool on all three models we test, including one from a different vendor. All code, per-call outputs, and pre-registered decision rules will be released upon acceptance.

Related papers

• K-Bench: measuring model performance on real scientific agent requests [0.0]
  K-Bench 01 is an evaluation built from first-turn requests sampled from live user traffic on K-Dense Web.<n>Three blinded language-model judges scored every run against an eight-dimension rubric.<n>Across all 39,934 scored judgments, 47.6% fall below the 8-point threshold.
  arXiv  Detail & Related papers  (2026-08-21T20:06:08Z)
• Decorrelation Is Not Complementarity: Skill, Not Lineage, Governs Trusted-Monitor Ensembles [0.0]
  We study 24 open-weight monitors spanning nine pretraining lineages and a 29x range of detection skill.<n>At matched member capability, cross-lineage panels detect no better.<n>No correlation-weighted selection beats picking the single best monitor out of sample.
  arXiv  Detail & Related papers  (2026-08-17T07:10:56Z)
• Deployment Decision Reliability: A Generalizability-Theory Framework for Sizing Long-Horizon Agent Evaluations [0.0]
  We show, across three open agent-trace benchmarks, that the agent main effect accounts for less than 3% of total variance in every dataset and check type.<n>We arrive at this through a four-facet Generalizability Theory variance decomposition, fit with three estimators.<n>We package these into Deployment Decision Reliability (DDR), a one-page reporting discipline that turns the variance-component table into five decisions an enterprise buyer can defend.
  arXiv  Detail & Related papers  (2026-08-11T18:16:51Z)
• On-Policy Self-Distillation without Any Supervision [47.315224514076334]
  We show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency.<n>We propose unsupervised on-policy self-distillation (U-OPSD)
  arXiv  Detail & Related papers  (2026-08-06T17:18:23Z)
• DarwinX: Evolving Agent Harnesses Through Natural Selection [48.64258219266629]
  DarwinX treats self-evolution as selection over a population of harnesses with the model frozen.<n>General agent competence, not benchmark-specific patches, so it survives changes of task, verifier, and base model.
  arXiv  Detail & Related papers  (2026-07-31T05:34:02Z)
• When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Models [0.0]
  Multi-model LLM systems are used to beat single-model accuracy.<n>We show that their gain is capped by a quantity the field rarely reports.
  arXiv  Detail & Related papers  (2026-06-25T17:06:06Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks [47.027290803102666]
  We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation.<n>A pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark.
  arXiv  Detail & Related papers  (2026-06-02T13:41:43Z)
• Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels [0.0]
  LLM-as-a-judge panels aggregate votes from multiple models.<n>We quantify how far their reliability falls short of the independent-voting ideal.
  arXiv  Detail & Related papers  (2026-05-28T11:48:17Z)
• Representational Collapse in Multi-Agent LLM Committees: Measurement and Diversity-Aware Consensus [0.0]
  Multi-agent LLM committees replicate the same model under different role prompts and aggregate outputs by majority vote.<n>We embed each agent's chain-of-thought rationale and measure pairwise similarity across 100 GSM8K questions with three Qwen2.5-14B agents.<n>DALC, a training-free consensus protocol that computes diversity weights from embedding geometry, reaches 87% on GSM8K versus 84% for self-consistency at 26% lower token cost.
  arXiv  Detail & Related papers  (2026-04-04T17:30:23Z)
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild [57.272096543738336]
  We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. We have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs.
  arXiv  Detail & Related papers  (2024-06-07T09:15:44Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.