Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards

• URL: http://arxiv.org/abs/2604.21769v1
• Date: Thu, 23 Apr 2026 15:28:32 GMT
• Title: Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboards
• Abstract summary: LLM leaderboards are widely used to compare models and guide deployment decisions.<n>However, leaderboard rankings are shaped by evaluation priorities set by benchmark designers.<n>A single aggregate score often obscures how models behave across different prompt types and compositions.
• Score: 9.132090357588686
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM leaderboards are widely used to compare models and guide deployment decisions. However, leaderboard rankings are shaped by evaluation priorities set by benchmark designers, rather than by the diverse goals and constraints of actual users and organizations. A single aggregate score often obscures how models behave across different prompt types and compositions. In this work, we conduct an in-depth analysis of the dataset used in the LMArena (formerly Chatbot Arena) benchmark and investigate this evaluation challenge by designing an interactive visualization interface as a design probe. Our analysis reveals that the dataset is heavily skewed toward certain topics, that model rankings vary across prompt slices, and that preference-based judgments are used in ways that blur their intended scope. Building on this analysis, we introduce a visualization interface that allows users to define their own evaluation priorities by selecting and weighting prompt slices and to explore how rankings change accordingly. A qualitative study suggests that this interactive approach improves transparency and supports more context-specific model evaluation, pointing toward alternative ways to design and use LLM leaderboards.

Related papers

• A Consensus-Based Framework for Relative Preference Evaluation of Large Language Models [0.0]
  This paper introduces a consensus-based evaluation framework that measures relative preference among model-generated responses rather than absolute correctness.<n>We conduct a controlled study using five state-of-the-art LLMs across multiple domains, including programming, general knowledge, safety, logical reasoning, and mathematics.<n>Scores are aggregated into a Relative Intelligence Index (RII), representing how frequently a model's responses are preferred by other models.
  arXiv  Detail & Related papers  (2026-07-19T08:59:19Z)
• Preference-Aware Rubric Learning for Personalized Evaluation [59.539429430690156]
  Existing evaluation methods fail to capture user-specific preferences embedded in long-term interaction histories.<n>We propose Personalized Evaluation as Learning, a paradigm that formulates personalized evaluation as a learning problem rather than a static judgment.<n>Experiments show that PARL consistently induces high-fidelity rubrics that reliably identify user-aligned responses and generalize across users.
  arXiv  Detail & Related papers  (2026-05-29T17:00:55Z)
• Personalized Benchmarking: Evaluating LLMs by Individual Preferences [16.446687355628566]
  Current benchmarks average preferences across all users to compute aggregate ratings.<n>Since users have varying preferences in different contexts, we call for personalized benchmarks that rank models according to individual needs.<n>We compute personalized model rankings using ELO ratings and Bradley-Terry coefficients for 115 active Arena users.
  arXiv  Detail & Related papers  (2026-04-21T00:40:47Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• MLLM as a UI Judge: Benchmarking Multimodal LLMs for Predicting Human Perception of User Interfaces [97.62557395494962]
  We use crowdsourcing to benchmark GPT-4o, Claude, and Llama across 30 interfaces.<n>Our results show that MLLMs approximate human preferences on some dimensions but diverge on others.
  arXiv  Detail & Related papers  (2025-10-09T20:00:41Z)
• Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic [4.087884819027264]
  This study applies BERTopic to the lmsys-chat-1m dataset, a multilingual conversational corpus built from head-to-head evaluations of large language models (LLMs)<n>The main objective is uncovering thematic patterns in these conversations and examining their relation to user preferences.<n>We analysed relationships between topics and model preferences to identify trends in model-topic alignment.
  arXiv  Detail & Related papers  (2025-10-08T21:13:44Z)
• LOTUS: A Leaderboard for Detailed Image Captioning from Quality to Societal Bias and User Preferences [91.13704541413551]
  LOTUS is a leaderboard for evaluating detailed captions.<n>It comprehensively evaluates various aspects, including caption quality.<n>It enables preference-oriented evaluations by tailoring criteria to diverse user preferences.
  arXiv  Detail & Related papers  (2025-07-25T15:12:42Z)
• SCAN: Structured Capability Assessment and Navigation for LLMs [54.54085382131134]
  textbfSCAN (Structured Capability Assessment and Navigation) is a practical framework that enables detailed characterization of Large Language Models.<n>SCAN incorporates four key components:.<n>TaxBuilder, which extracts capability-indicating tags from queries to construct a hierarchical taxonomy;.<n>RealMix, a query synthesis and filtering mechanism that ensures sufficient evaluation data for each capability tag;.<n>A PC$2$-based (Pre-Comparison-derived Criteria) LLM-as-a-Judge approach achieves significantly higher accuracy compared to classic LLM-as-a-Judge method
  arXiv  Detail & Related papers  (2025-05-10T16:52:40Z)
• Improving LLM Leaderboards with Psychometrical Methodology [0.0]
  The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance.<n>These benchmarks resemble human tests and surveys, as they consist of questions designed to measure emergent properties in the cognitive behavior of these systems.<n>However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined.
  arXiv  Detail & Related papers  (2025-01-27T21:21:46Z)
• Towards Personalized Evaluation of Large Language Models with An Anonymous Crowd-Sourcing Platform [64.76104135495576]
  We propose a novel anonymous crowd-sourcing evaluation platform, BingJian, for large language models. Through this platform, users have the opportunity to submit their questions, testing the models on a personalized and potentially broader range of capabilities.
  arXiv  Detail & Related papers  (2024-03-13T07:31:20Z)
• When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards [9.751405901938895]
  We show that under existing leaderboards, the relative performance of LLMs is highly sensitive to minute details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions.
  arXiv  Detail & Related papers  (2024-02-01T19:12:25Z)
• FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets [69.91340332545094]
  We introduce FLASK, a fine-grained evaluation protocol for both human-based and model-based evaluation. We experimentally observe that the fine-graininess of evaluation is crucial for attaining a holistic view of model performance.
  arXiv  Detail & Related papers  (2023-07-20T14:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.