BenchScope: How Many Independent Signals Does Your Benchmark Provide?

• URL: http://arxiv.org/abs/2603.29357v1
• Date: Tue, 31 Mar 2026 07:28:41 GMT
• Title: BenchScope: How Many Independent Signals Does Your Benchmark Provide?
• Abstract summary: We introduce Effective Dimensionality (ED), a fast, population-conditional upper-bound diagnostic of measurement breadth.<n>ED is applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations.<n>We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI evaluation suites often report many scores without checking whether those scores carry independent information. We introduce Effective Dimensionality (ED), the participation ratio of a centered benchmark-score spectrum, as a fast, population-conditional upper-bound diagnostic of measurement breadth. Applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations, ED reveals substantial redundancy: the six-score Open LLM Leaderboard behaves like roughly two effective measurement axes (ED = 1.7), BBH and MMLU-Pro are near-interchangeable (rho = 0.96, stable across seven subpopulations), and measurement breadth varies more than 20x across current benchmarks. We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components, monitor performance-conditional compression, and guide benchmark maintenance. Because binary spectra overestimate absolute latent dimensionality, we interpret ED as a screening statistic rather than a literal factor count and complement it with null, reliability, and saturation analyses. We provide a 22-benchmark reference atlas and a four-step diagnostic workflow that benchmark maintainers can run with a score matrix and a few lines of code.

Related papers

• The Double Measurement Confound in Agent Benchmarks: De-Scaffolding, Ground-Truth Scoring, and Reliability Beyond the Mean [2.822987192839875]
  Agent benchmarks are increasingly used to compare large language models (LLMs) and guide deployment decisions.<n>We identify a double measurement confound: execution-critical decisions are performed by a fixed scaffold instead of the model, while the scorer evaluates outputs using criteria that may not reflect task correctness.<n>We unify these issues within a measurement-theoretic framework that characterizes when benchmark scores can be interpreted as evidence of model capability.
  arXiv  Detail & Related papers  (2026-09-06T21:23:11Z)
• VISTA: Dense Multi-Label Classroom Coding with Vision-Language Models [0.0]
  We argue that multimodal benchmarking benefits from methods taken from research communities that have already invested in strategies to ensure reliability.<n>We recast COPUS as a video benchmark for multimodal foundation models, where it provides a dense set of structured labels.<n>We propose VISTA, a baseline that runs MiniCPM-V-4.5 over a dense sliding window, refines its per-window outputs with a lightweight multi-layer perceptron (MLP) head trained on top of the frozen backbone.
  arXiv  Detail & Related papers  (2026-09-03T23:13:56Z)
• A Statistical Audit of Physical AI Benchmark Redundancy [0.0]
  Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse.<n>We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density.<n>We measure how much information the benchmarks share and show quantitative evidence of Redundancy.
  arXiv  Detail & Related papers  (2026-08-26T15:54:15Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• BenchBench: Benchmarking Automated Benchmark Generation [10.44497524694021]
  BenchBench is a pipeline and dataset for benchmarking automated benchmark generation.<n>We generate 16.7K items, retain 15K core items post-filtering, and produce 152K graded model--item responses.
  arXiv  Detail & Related papers  (2026-03-21T13:05:32Z)
• DEP: A Decentralized Large Language Model Evaluation Protocol [51.3646001384887]
  Decentralized Evaluation Protocol (DEP) is a decentralized yet unified and standardized evaluation framework.<n>By decoupling users, LLMs, and benchmarks, DEP enables modular, plug-and-play evaluation.<n>We develop DEP Toolkit, a protocol-compatible toolkit that supports features such as breakpoint resume, concurrent requests, and congestion control.
  arXiv  Detail & Related papers  (2026-03-01T16:10:16Z)
• Fantastic Bugs and Where to Find Them in AI Benchmarks [28.604919035475188]
  We introduce a framework for systematic benchmark revision that leverages statistical analysis of response patterns to flag potentially invalid questions.<n>Our approach builds on a core assumption commonly used in AI evaluations that the mean score sufficiently summarizes model performance.<n>Across nine widely used benchmarks, our method guides expert review to identify problematic questions with up to 84% precision.
  arXiv  Detail & Related papers  (2025-11-20T22:49:21Z)
• Benchmark Designers Should "Train on the Test Set" to Expose Exploitable Non-Visual Shortcuts [49.99400612296149]
  We find that models can ace many benchmarks without strong visual understanding.<n>This is especially problematic for vision-centric benchmarks that are meant to require visual inputs.<n>We adopt a diagnostic principle for benchmark design: if a benchmark can be gamed, it will be gamed.
  arXiv  Detail & Related papers  (2025-11-06T18:43:21Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs [60.25940747590386]
  We propose How2Bench, which is comprised of a 55-criteria checklist as a set of guidelines to govern the development of code-related benchmarks comprehensively.<n>We profiled 274 benchmarks released within the past decade and found concerning issues.<n>Nearly 70% of the benchmarks did not take measures for data quality assurance; over 10% did not even open source or only partially open source.
  arXiv  Detail & Related papers  (2025-01-18T09:51:57Z)
• MMIE: Massive Multimodal Interleaved Comprehension Benchmark for Large Vision-Language Models [71.36392373876505]
  We introduce MMIE, a large-scale benchmark for evaluating interleaved multimodal comprehension and generation in Large Vision-Language Models (LVLMs)<n>MMIE comprises 20K meticulously curated multimodal queries, spanning 3 categories, 12 fields, and 102 subfields, including mathematics, coding, physics, literature, health, and arts.<n>It supports both interleaved inputs and outputs, offering a mix of multiple-choice and open-ended question formats to evaluate diverse competencies.
  arXiv  Detail & Related papers  (2024-10-14T04:15:00Z)
• metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models [5.972993094932516]
  Large Language Models (LLMs) vary in their abilities on a range of tasks.<n>We show that there exists a small set of common underlying abilities that these benchmarks measure.<n>We distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined.
  arXiv  Detail & Related papers  (2024-07-04T17:57:38Z)
• ACCORD: Closing the Commonsense Measurability Gap [16.572584339052753]
  ACCORD is a framework for disentangling the commonsense grounding and reasoning abilities of large language models (LLMs)<n>It introduces formal elements to commonsense reasoning to explicitly control and quantify reasoning complexity beyond the typical 1 or 2 hops.<n>It can automatically generate benchmarks of arbitrary reasoning complexity, and so it scales with future LLM improvements.
  arXiv  Detail & Related papers  (2024-06-04T22:08:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.