The Trust Paradox: How CS Researchers Engage LLM Leaderboards

• URL: http://arxiv.org/abs/2605.28966v1
• Date: Wed, 27 May 2026 18:09:26 GMT
• Title: The Trust Paradox: How CS Researchers Engage LLM Leaderboards
• Abstract summary: Leaderboards rank AI models using standardized benchmarks, but how they shape researchers' actual practice remains empirically uncharted.<n>We find a near-universal paradox of pragmatic skepticism: while participants expressed deep distrust of leaderboard rankings, they continued to use them as rough decision-making aids.<n>We translate these findings into concrete design recommendations that align evaluation infrastructure with how researchers actually use it.
• Score: 49.627487948130444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) leaderboards rank AI models using standardized benchmarks and have become highly visible across computer science, despite known limitations in their reliability and robustness. Yet how they shape researchers' actual practice remains empirically uncharted. We address this gap through semi-structured interviews with eight researchers across four computer science subfields, analyzed using reflexive thematic analysis. We find a near-universal paradox of pragmatic skepticism: while participants expressed deep distrust of leaderboard rankings, they continued to use them as rough decision-making aids. Peer networks, not leaderboards, emerged as the primary model selection mechanism, and arena-based (human-voting) leaderboards were consistently preferred over static benchmark leaderboards. Leaderboard influence varied sharply across subfields, revealing that disciplinary culture, not individual attitudes, mediates engagement; for instance, NLP researchers faced state-of-the-art comparison pressure while HCI and Systems/Privacy researchers reported none. Across these differences, however, participants converged on cost transparency as the most demanded missing feature (seven of eight). We translate these findings into concrete design recommendations that align evaluation infrastructure with how researchers actually use it, such as task-specific score breakdowns, cost integration, and voter-demographic disclosure.

Related papers

• Ideation Arena: Evaluating LLM Generated Research Ideas with Battle-style Human Expert Assessment [18.14707450829467]
  Ideation Arena is a battle style platform that evaluates research ideas through pairwise human assessment.<n>We collect over 6,000 double blind pairwise comparisons from 105 active computer science researchers.<n>We construct an Elo rating leaderboard of proposal-stage expert preferences in computer science.
  arXiv  Detail & Related papers  (2026-08-30T10:10:10Z)
• OmniOPSD: Rationale-Privileged On-Policy Self-Distillation for Affective Computing [62.62736813348113]
  Reinforcement learning for large language models is often hindered by severe reward sparsity in complex reasoning tasks.<n>We propose OmniOPSD, a framework that uses frontier-generated rationales as teacher-side privileged evidence.<n>Experiments on MER-UniBench show that OmniOPSD achieves state-of-the-art performance with an average score of $84.19$.
  arXiv  Detail & Related papers  (2026-06-14T16:51:22Z)
• Not My Truce: Personality Differences in AI-Mediated Workplace Negotiation [15.57105036031543]
  We compare theory-driven AI (Trucey), general-purpose AI (Control-AI), and a traditional negotiation handbook (Control-NoAI)<n>Resilient workers achieved broad psychological gains primarily from the handbook.<n>Overcontrolled workers showed outcome-specific improvements with theory-driven AI.<n>Undercontrolled workers exhibited minimal effects despite engaging with the frameworks.
  arXiv  Detail & Related papers  (2026-04-01T04:26:26Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• An External Fairness Evaluation of LinkedIn Talent Search [55.18656975953939]
  We conduct an independent, third-party audit for bias of LinkedIn's Talent Search ranking system.<n>We focus on potential ranking bias across two attributes: gender and race.<n>Our analysis reveals an under-representation of minority groups in early ranks.
  arXiv  Detail & Related papers  (2025-11-13T19:10:49Z)
• MoVa: Towards Generalizable Classification of Human Morals and Values [57.93595662296688]
  MoVa is a well-documented suite of resources for generalizable classification of human morals and values.<n>The data and methods in MoVa can facilitate many fine-grained interpretations of human and machine communication.
  arXiv  Detail & Related papers  (2025-09-29T02:56:27Z)
• Mitigating Subgroup Disparities in Multi-Label Speech Emotion Recognition: A Pseudo-Labeling and Unsupervised Learning Approach [53.824673312331626]
  Implicit Demography Inference (IDI) module uses k-means clustering to mitigate bias in Speech Emotion Recognition (SER)<n>Experiments show that pseudo-labeling IDI reduces subgroup disparities, improving fairness metrics by over 28%.<n>Unsupervised IDI yields more than a 4.6% improvement in fairness metrics with a drop of less than 3.6% in SER performance.
  arXiv  Detail & Related papers  (2025-05-20T14:50:44Z)
• Improving LLM Leaderboards with Psychometrical Methodology [0.0]
  The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance.<n>These benchmarks resemble human tests and surveys, as they consist of questions designed to measure emergent properties in the cognitive behavior of these systems.<n>However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined.
  arXiv  Detail & Related papers  (2025-01-27T21:21:46Z)
• More than Marketing? On the Information Value of AI Benchmarks for Practitioners [42.73526862595375]
  In academia, public benchmarks were generally viewed as suitable measures for capturing research progress.<n>In product and policy, benchmarks were often found to be inadequate for informing substantive decisions.<n>We conclude that effective benchmarks should provide meaningful, real-world evaluations, incorporate domain expertise, and maintain transparency in scope and goals.
  arXiv  Detail & Related papers  (2024-12-07T03:35:39Z)
• Efficient Performance Tracking: Leveraging Large Language Models for Automated Construction of Scientific Leaderboards [67.65408769829524]
  Scientific leaderboards are standardized ranking systems that facilitate evaluating and comparing competitive methods. The exponential increase in publications has made it infeasible to construct and maintain these leaderboards manually. automatic leaderboard construction has emerged as a solution to reduce manual labor.
  arXiv  Detail & Related papers  (2024-09-19T11:12:27Z)
• Self-Improving-Leaderboard(SIL): A Call for Real-World Centric Natural Language Processing Leaderboards [5.919860270977038]
  We argue that evaluation on a given test dataset is just one of many performance indications of the model. We propose a new paradigm of leaderboard systems that addresses these issues of current leaderboard system.
  arXiv  Detail & Related papers  (2023-03-20T06:13:03Z)
• Automated Mining of Leaderboards for Empirical AI Research [0.0]
  This study presents a comprehensive approach for generating Leaderboards for knowledge-graph-based scholarly information organization. Specifically, we investigate the problem of automated Leaderboard construction using state-of-the-art transformer models, viz. Bert, SciBert, and XLNet. As a result, a vast share of empirical AI research can be organized in the next-generation digital libraries as knowledge graphs.
  arXiv  Detail & Related papers  (2021-08-31T10:00:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.