Strategic Candidacy in Generative AI Arenas

• URL: http://arxiv.org/abs/2603.26891v1
• Date: Fri, 27 Mar 2026 18:12:58 GMT
• Title: Strategic Candidacy in Generative AI Arenas
• Abstract summary: We propose a new mechanism for ranking models from pairwise comparisons, called YouRankWeRank-YRWR.<n>It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality.<n>We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once.
• Score: 5.648790815017435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI arenas, which rank generative models from pairwise preferences of users, are a popular method for measuring the relative performance of models in the course of their organic use. Because rankings are computed from noisy preferences, there is a concern that model producers can exploit this randomness by submitting many models (e.g., multiple variants of essentially the same model) and thereby artificially improve the rank of their top models. This can lead to degradations in the quality, and therefore the usefulness, of the ranking. In this paper, we begin by establishing, both theoretically and in simulations calibrated to data from the platform Arena (formerly LMArena, Chatbot Arena), conditions under which producers can benefit from submitting clones when their goal is to be ranked highly. We then propose a new mechanism for ranking models from pairwise comparisons, called You-Rank-We-Rank (YRWR). It requires that producers submit rankings over their own models and uses these rankings to correct statistical estimates of model quality. We prove that this mechanism is approximately clone-robust, in the sense that a producer cannot improve their rank much by doing anything other than submitting each of their unique models exactly once. Moreover, to the extent that model producers are able to correctly rank their own models, YRWR improves overall ranking accuracy. In further simulations, we show that indeed the mechanism is approximately clone-robust and quantify improvements to ranking accuracy, even under producer misranking.

Related papers

• Dynamically Allocating Evaluation Effort for Model Ranking [60.00410111594536]
  We formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms.<n>We show that it improves discrimination between top-performing models.<n>This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
  arXiv  Detail & Related papers  (2026-08-04T10:33:04Z)
• UniRank: Benchmarking Ranking Models for Unified Sequential Modeling and Feature Interaction [10.489900944065083]
  This paper proposes UniRank, an open benchmark for ranking models that unify sequential modeling and feature interaction.<n>We benchmark 15 representative unified ranking models on five large-scale public datasets from short-video, advertising, and e-commerce platforms.<n>UniRank provides a reproducible basis for comparing unified ranking models, studying scaling laws under limited compute, and narrowing the gap between academic and industrial ranking research.
  arXiv  Detail & Related papers  (2026-07-22T10:20:40Z)
• Explanation Quality Assessment as Ranking with Listwise Rewards [21.107094956399262]
  We reformulate explanation quality assessment as a ranking problem rather than a generation problem.<n>We construct per-instance candidate sets with graded quality levels and train reward models to discriminate among multiple candidate explanations.<n>When used as rewards in policy optimization, ranking-based scores enable stable convergence in settings where regression-based rewards fail entirely.
  arXiv  Detail & Related papers  (2026-04-27T08:35:30Z)
• ReasonRank: Empowering Passage Ranking with Strong Reasoning Ability [83.16850534680505]
  We propose an automated reasoning-intensive training data synthesis framework.<n>A self-consistency data filtering mechanism is designed to ensure the data quality.<n>Our trained reasoning-intensive reranker textbfReasonRank achieves state-of-the-art (SOTA) performance 40.6 on the BRIGHT leaderboard.
  arXiv  Detail & Related papers  (2025-08-09T17:26:18Z)
• CHARM: Calibrating Reward Models With Chatbot Arena Scores [31.599659350165354]
  Reward models (RMs) play a crucial role in Reinforcement Learning from Human Feedback by serving as proxies for human preferences in aligning large language models.<n>We identify a model preference bias in RMs, where they systematically assign disproportionately high scores to responses from certain policy models.<n>To address this issue, we propose a calibration method named CHatbot Arena Reward Modeling (CHARM) that leverages Elo scores from the Arena leaderboard to mitigate RM overvaluation.
  arXiv  Detail & Related papers  (2025-04-14T09:51:09Z)
• K-Sort Arena: Efficient and Reliable Benchmarking for Generative Models via K-wise Human Preferences [30.744662265421788]
  Arena platform, which gathers user votes on model comparisons, can rank models with human preferences.<n>We introduce K-Sort Arena, an efficient and reliable platform based on a key insight: images and videos possess higher perceptual intuitiveness than texts.<n>In our experiments, K-Sort Arena exhibits 16.3x faster convergence compared to the widely used ELO algorithm.
  arXiv  Detail & Related papers  (2024-08-26T17:58:20Z)
• Investigating the Robustness of Counterfactual Learning to Rank Models: A Reproducibility Study [71.04084063541777]
  Counterfactual learning to rank has attracted extensive attention in the IR community.<n>Models can be theoretically unbiased when the user behavior assumption is correct and the propensity estimation is accurate.<n>Their effectiveness is usually empirically evaluated via simulation-based experiments due to a lack of widely available, large-scale, real click logs.
  arXiv  Detail & Related papers  (2024-04-04T10:54:38Z)
• Learning Rich Rankings [7.940293148084844]
  We develop a contextual repeated selection (CRS) model to bring a natural multimodality and richness to the rankings space. We provide theoretical guarantees for maximum likelihood estimation under the model through structure-dependent tail risk and expected risk bounds. We also furnish the first tight bounds on the expected risk of maximum likelihood estimators for the multinomial logit (MNL) choice model and the Plackett-Luce (PL) ranking model.
  arXiv  Detail & Related papers  (2023-12-22T21:40:57Z)
• COPR: Consistency-Oriented Pre-Ranking for Online Advertising [27.28920707332434]
  We introduce a consistency-oriented pre-ranking framework for online advertising. It employs a chunk-based sampling module and a plug-and-play rank alignment module to explicitly optimize consistency of ECPM-ranked results. When deployed in Taobao display advertising system, it achieves an improvement of up to +12.3% CTR and +5.6% RPM.
  arXiv  Detail & Related papers  (2023-06-06T09:08:40Z)
• The False Promise of Imitating Proprietary LLMs [158.65692029352584]
  An emerging method to cheaply improve a weaker language model is to finetune it on outputs from a stronger model. This approach looks to cheaply imitate the proprietary model's capabilities using a weaker open-source model. We first finetune a series of LMs that imitate ChatGPT using varying base model sizes. We then evaluate the models using crowd raters and canonical NLP benchmarks.
  arXiv  Detail & Related papers  (2023-05-25T05:00:12Z)
• Interpretable Learning-to-Rank with Generalized Additive Models [78.42800966500374]
  Interpretability of learning-to-rank models is a crucial yet relatively under-examined research area. Recent progress on interpretable ranking models largely focuses on generating post-hoc explanations for existing black-box ranking models. We lay the groundwork for intrinsically interpretable learning-to-rank by introducing generalized additive models (GAMs) into ranking tasks.
  arXiv  Detail & Related papers  (2020-05-06T01:51:30Z)
• AvgOut: A Simple Output-Probability Measure to Eliminate Dull Responses [97.50616524350123]
  We build dialogue models that are dynamically aware of what utterances or tokens are dull without any feature-engineering. The first model, MinAvgOut, directly maximizes the diversity score through the output distributions of each batch. The second model, Label Fine-Tuning (LFT), prepends to the source sequence a label continuously scaled by the diversity score to control the diversity level. The third model, RL, adopts Reinforcement Learning and treats the diversity score as a reward signal.
  arXiv  Detail & Related papers  (2020-01-15T18:32:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.