Investigating Non-Transitivity in LLM-as-a-Judge

• URL: http://arxiv.org/abs/2502.14074v2
• Date: Thu, 06 Mar 2025 06:32:54 GMT
• Title: Investigating Non-Transitivity in LLM-as-a-Judge
• Authors: Yi Xu, Laura Ruis, Tim Rocktäschel, Robert Kirk,
• Abstract summary: We investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings.<n>To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments.
• Score: 24.358802214160697
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic evaluation methods based on large language models (LLMs) are emerging as the standard tool for assessing the instruction-following abilities of LLM-based agents. The most common method in this paradigm, pairwise comparisons with a baseline model, critically depends on the assumption of transitive preferences. However, the validity of this assumption remains largely unexplored. In this study, we investigate the presence of non-transitivity within the AlpacaEval framework and analyze its effects on model rankings. We find that LLM judges exhibit non-transitive preferences, leading to rankings that are sensitive to the choice of the baseline model. To mitigate this issue, we show that round-robin tournaments combined with Bradley-Terry models of preference can produce more reliable rankings. Notably, our method increases both the Spearman correlation and the Kendall correlation with Chatbot Arena (95.0% -> 96.4% and 82.1% -> 86.3% respectively). To address the computational cost of round-robin tournaments, we propose Swiss-Wise Iterative Matchmaking (Swim) tournaments, using a dynamic matching strategy to capture the benefits of round-robin tournaments while maintaining computational efficiency.

Related papers

• CHARM: Calibrating Reward Models With Chatbot Arena Scores [31.599659350165354]
  Reward models (RMs) play a crucial role in Reinforcement Learning from Human Feedback by serving as proxies for human preferences in aligning large language models. We identify a model preference bias in RMs, where they systematically assign disproportionately high scores to responses from certain policy models. To address this issue, we propose a calibration method named CHatbot Arena Reward Modeling (CHARM) that leverages Elo scores from the Arena leaderboard to mitigate RM overvaluation.
  arXiv  Detail & Related papers  (2025-04-14T09:51:09Z)
• A Statistical Framework for Ranking LLM-Based Chatbots [57.59268154690763]
  We propose a statistical framework that incorporates key advancements to address specific challenges in pairwise comparison analysis. First, we introduce a factored tie model that enhances the ability to handle groupings of human-judged comparisons. Second, we extend the framework to model covariance tiers between competitors, enabling deeper insights into performance relationships. Third, we resolve optimization challenges arising from parameter non-uniqueness by introducing novel constraints.
  arXiv  Detail & Related papers  (2024-12-24T12:54:19Z)
• Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning [15.776175440446414]
  We introduce Dr.SoW (Density Ratio of Strong over Weak) a cost-effective method that eliminates the reliance for human annotation.<n>Dr.SoW uses the log-density ratio between a better-aligned and a less-aligned LLM as a reward signal.<n>We preference-tune Llama-3-8B-Instruct using data annotated by Dr.SoW.
  arXiv  Detail & Related papers  (2024-11-04T18:54:39Z)
• Diverging Preferences: When do Annotators Disagree and do Models Know? [92.24651142187989]
  We develop a taxonomy of disagreement sources spanning 10 categories across four high-level classes. We find that the majority of disagreements are in opposition with standard reward modeling approaches. We develop methods for identifying diverging preferences to mitigate their influence on evaluation and training.
  arXiv  Detail & Related papers  (2024-10-18T17:32:22Z)
• Constructing Domain-Specific Evaluation Sets for LLM-as-a-judge [15.980606104936365]
  Large Language Models (LLMs) have revolutionized the landscape of machine learning, yet current benchmarks often fall short in capturing the diverse behavior of these models in real-world applications. Existing frameworks like Alpaca-Eval 2.0 LC citedubois2024lengthcontrolledalpacaevalsimpleway and Arena-Hard v0.1 citeli2024crowdsourced are limited by their focus on general-purpose queries and lack of diversity across domains such as law, medicine, and multilingual contexts. We introduce a novel data pipeline that curates, domain-specific evaluation sets tailored for LLM-as
  arXiv  Detail & Related papers  (2024-08-16T15:41:43Z)
• Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning [55.65738319966385]
  We propose a novel online algorithm, iterative Nash policy optimization (INPO) Unlike previous methods, INPO bypasses the need for estimating the expected win rate for individual responses. With an LLaMA-3-8B-based SFT model, INPO achieves a 42.6% length-controlled win rate on AlpacaEval 2.0 and a 37.8% win rate on Arena-Hard.
  arXiv  Detail & Related papers  (2024-06-30T08:00:34Z)
• A LLM-Based Ranking Method for the Evaluation of Automatic Counter-Narrative Generation [14.064465097974836]
  This paper proposes a novel approach to evaluate Counter Narrative (CN) generation using a Large Language Model (LLM) as an evaluator. We show that traditional automatic metrics correlate poorly with human judgements and fail to capture the nuanced relationship between generated CNs and human perception.
  arXiv  Detail & Related papers  (2024-06-21T15:11:33Z)
• Dissecting Human and LLM Preferences [80.55271307662365]
  We find that humans are less sensitive to errors, favor responses that support their stances, and show clear dislike when models admit their limits. advanced LLMs like GPT-4-Turbo emphasize correctness, clarity, and harmlessness more. We show that preference-based evaluation can be intentionally manipulated.
  arXiv  Detail & Related papers  (2024-02-17T14:34:31Z)
• A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models [35.17291316942284]
  We propose a novel zero-shot document ranking approach based on Large Language Models (LLMs): the Setwise prompting approach. Our approach complements existing prompting approaches for LLM-based zero-shot ranking: Pointwise, Pairwise, and Listwise.
  arXiv  Detail & Related papers  (2023-10-14T05:20:02Z)
• PRD: Peer Rank and Discussion Improve Large Language Model based Evaluations [10.709365940160685]
  Modern large language models (LLMs) are hard to evaluate and compare automatically. We propose a peer rank (PR) algorithm that takes into account each peer LLM's pairwise preferences of all answer pairs. We find that our approaches achieve higher accuracy and align better with human judgments.
  arXiv  Detail & Related papers  (2023-07-06T04:05:44Z)
• Exploring validation metrics for offline model-based optimisation with diffusion models [50.404829846182764]
  In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation.
  arXiv  Detail & Related papers  (2022-11-19T16:57:37Z)
• Towards Model-Agnostic Post-Hoc Adjustment for Balancing Ranking Fairness and Algorithm Utility [54.179859639868646]
  Bipartite ranking aims to learn a scoring function that ranks positive individuals higher than negative ones from labeled data. There have been rising concerns on whether the learned scoring function can cause systematic disparity across different protected groups. We propose a model post-processing framework for balancing them in the bipartite ranking scenario.
  arXiv  Detail & Related papers  (2020-06-15T10:08:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.