Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings

• URL: http://arxiv.org/abs/2606.09409v1
• Date: Mon, 08 Jun 2026 12:26:54 GMT
• Title: Correct Looks Better: Pairwise Comparisons Reveal Accuracy Rankings
• Abstract summary: We show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison.<n>We find that style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct.
• Score: 30.711773754965986
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Pairwise comparisons combined with aggregation methods like Elo have become central to evaluating generative models, yet concerns remain that they reward superficial stylistic cues or display judge biases. In a more positive turn, we show that model rankings from pairwise comparisons strongly agree with ground-truth-based accuracy rankings when such ground truth is available for comparison. By converting five well-known benchmarks into free-form generative evaluations, we find that Elo rankings achieve a Spearman correlation above 0.9 with accuracy rankings and substantially outperform direct evaluation when the judge is weak. Furthermore, style and judge bias have only minor effects on model rankings, despite most judgments occurring on pairs where both candidate answers are correct (or incorrect). On such pairs, we find that repetition after the final answer (echo) is a causal driver of judge preference.

Related papers

• Ask the Right Comparison:Bias-Aware Bayesian Active Top-$k$ Ranking with LLM Judges [31.082191748525137]
  Large language models (LLMs) are increasingly used as cheap, scalable judges that compare candidate outputs pairwise.<n>We study the practical goal of identifying the topk items under a fixed comparison budget.<n>We introduce a topk-aware active acquisition rule that chooses the next comparison to maximally reduce uncertainty.
  arXiv  Detail & Related papers  (2026-07-02T12:39:30Z)
• Reasoning Arena: Trace Tournaments When Verifiable Rewards Fall Short [51.667769734342635]
  Reinforcement learning with verifiable rewards (RLVR) has become a leading paradigm for improving the reasoning ability of large language models.<n>We propose Reasoning Arena, an adaptive training framework that routes such non-diverse reward groups to a judge system instead of discarding them.<n>We show that Reasoning Arena consistently outperforms the RLVR baseline by 7.6% on average in competition mathematics and coding benchmarks.
  arXiv  Detail & Related papers  (2026-06-08T11:57:17Z)
• The Attention Market: Interpreting Online Fair Re-ranking as Manifold Optimization under Walrasian Equilibrium [63.76805218520176]
  We reformulate fair re-ranking within an attentional market framework governed by a Walrasian Equilibrium.<n>We propose ManifoldRank, an efficient online fair re-ranking algorithm.
  arXiv  Detail & Related papers  (2026-04-28T12:43:57Z)
• New insights into Elo algorithm for practitioners and statisticians [0.8122270502556374]
  We show that estimation noise forces a principled decoupling between the model used for ranking and the model used for prediction.<n>We find that the ranking had not converged for the vast majority of national teams.
  arXiv  Detail & Related papers  (2026-04-04T19:38:28Z)
• Does Reasoning Make Search More Fair? Comparing Fairness in Reasoning and Non-Reasoning Rerankers [56.54960821093771]
  We conduct the first systematic comparison of fairness between reasoning and non-reasoning rerankers.<n>Our findings demonstrate reasoning neither improve nor harm fairness compared to non-reasoning approaches.
  arXiv  Detail & Related papers  (2026-03-11T02:01:24Z)
• PeerRank: Autonomous LLM Evaluation Through Web-Grounded, Bias-Controlled Peer Review [1.2178992475191557]
  We introduce PeerRank, a fully autonomous end-to-end evaluation framework.<n>Models generate evaluation tasks, answer them with category-scoped live web grounding.<n>PeerRank treats evaluation as a multi-agent process where each model participates symmetrically as task designer, respondent, and evaluator.
  arXiv  Detail & Related papers  (2026-02-01T06:01:28Z)
• What Is the Optimal Ranking Score Between Precision and Recall? We Can Always Find It and It Is Rarely $F_1$ [17.215680052668244]
  We establish that $F_$-induced rankings are meaningful and define a shortest path between precision- and recall-induced rankings.<n>We provide theoretical tools and a closed-form expression to find the optimal value for $$ for any distribution or set of performances.
  arXiv  Detail & Related papers  (2025-11-27T13:29:50Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• UDA: Unsupervised Debiasing Alignment for Pair-wise LLM-as-a-Judge [23.497453639857852]
  We propose UDA, a framework that reduces inter-judge disagreement by dynamically adjusting the Elo rating system.<n>UDA operates in a fully unsupervised manner, guided solely by the objective of minimizing the dispersion among the Elo trajectories of all judges.<n>Experiments show that UDA significantly reduces the inter-judge rating standard deviation by up to 63.4% and improves the average correlation with human judgments by 24.7%.
  arXiv  Detail & Related papers  (2025-08-13T11:41:01Z)
• J4R: Learning to Judge with Equivalent Initial State Group Relative Policy Optimization [69.23273504123941]
  We train judges to be robust to positional biases that arise in more complex evaluation settings.<n>We introduce ReasoningJudgeBench, a benchmark that evaluates judges in diverse reasoning settings not covered by prior work.<n>We train Judge for Reasoning (J4R), a 7B judge trained with EIS-GRPO that outperforms GPT-4o and the next best small judge by 6.7% and 9%.
  arXiv  Detail & Related papers  (2025-05-19T16:50:35Z)
• Where Fact Ends and Fairness Begins: Redefining AI Bias Evaluation through Cognitive Biases [77.3489598315447]
  We argue that identifying the boundary between fact and fair is essential for meaningful fairness evaluation.<n>We introduce Fact-or-Fair, a benchmark with (i) objective queries aligned with descriptive, fact-based judgments, and (ii) subjective queries aligned with normative, fairness-based judgments.
  arXiv  Detail & Related papers  (2025-02-09T10:54:11Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• Fair Pairs: Fairness-Aware Ranking Recovery from Pairwise Comparisons [2.056289813004423]
  We introduce the problem of fairness-aware ranking recovery from pairwise comparisons. We propose a group-conditioned accuracy measure which quantifies fairness of rankings recovered from pairwise comparisons.
  arXiv  Detail & Related papers  (2024-08-23T12:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.