A Statistical Audit of Physical AI Benchmark Redundancy

• URL: http://arxiv.org/abs/2608.25940v2
• Date: Sat, 29 Aug 2026 14:08:12 GMT
• Title: A Statistical Audit of Physical AI Benchmark Redundancy
• Abstract summary: Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse.<n>We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density.<n>We measure how much information the benchmarks share and show quantitative evidence of Redundancy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse and the relationship between benchmarks unmeasured. We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density, combining scores from model cards and benchmark papers with our own evaluation runs under each benchmark's official protocol. We measure how much information the benchmarks share and show quantitative evidence of Redundancy. Redundancy affects reported rankings: collapsing the two substitute pairs into single columns moves 22 of 51 models by three or more places under an equally weighted average. We then select benchmarks greedily under a utility combining score dispersion with variance not explained by the already-selected set, and obtain a four-benchmark subset retaining 78.5\% of the utility of all 12, on which we fit a Bradley--Terry ranking. The procedure requires only benchmark-level scores with sufficient overlap and is not specific to physical AI.

Related papers

• Balance of Benchmarks: Semantic Density Reweighting for Benchmark Multiplicity and Task-Conditioned Evaluation [0.0]
  Balance of Benchmarks embeds benchmark descriptions and assigns each benchmark an inverse-density semantic weight.<n>A residual field uses the same geometry to condition model rankings on a task query.<n>BoB attains the highest mean profile and significantly outperforms nearest-neighbour, tuned top-(k), radius, and cluster-based aggregation.
  arXiv  Detail & Related papers  (2026-08-30T21:06:29Z)
• RepBench: Compiling Benchmarks into Capability Representations for Large Language Models [5.276992083399684]
  RepBench is a benchmark-grounded data layer for capability-aligned representation probing.<n>We crawl 13,427 benchmark papers yields a taxonomy of 182 capability clusters in 13 families; harvesting 353 public benchmark datasets yields 46,149 audited probe texts covering 94 capabilities.
  arXiv  Detail & Related papers  (2026-07-30T10:56:44Z)
• PACE: A Proxy for Agentic Capability Evaluation [60.3743414796937]
  We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations.<n>Experiments across 14 models, 4 agentic benchmarks, and 19 non-agentic benchmarks show that PACE-Bench predicts agentic scores with leave-one-out cross-validation (LOOCV) mean absolute error (MAE) under 4%, Spearman correlation above 0.80, and pairwise model-ranking accuracy around 85%, all at much less than 1% of the full agentic evaluation cost.
  arXiv  Detail & Related papers  (2026-07-02T10:59:03Z)
• Knowledge Index of Noah's Ark [63.143852586221534]
  We introduce KINA, an 899-item benchmark across 261 fine-grained disciplines.<n>We prove a bonus-on-bar tournament weakly FOSD-dominates flat payment in released-review quality.<n>The top model, Gemini-3.1-Pro-Preview, reaches 53.17%, followed by Claude-Opus-4.6 at 49.92% and GPT-5.4 at 48.55%.
  arXiv  Detail & Related papers  (2026-06-03T17:06:49Z)
• Putting HUMANS first: Efficient LAM Evaluation with Human Preference Alignment [53.72927532626824]
  We show that subsets of just 50 examples (0.3% of data) can achieve over 0.93 Pearson correlation with full benchmark scores.<n>To better predict preferences, we trained regression models on these selected subsets, achieving 0.98 correlation.<n>This demonstrates that in regression modeling, well-curated subsets outpredict the full benchmark, showing quality over quantity.
  arXiv  Detail & Related papers  (2026-04-20T00:57:31Z)
• When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation [80.66788281323414]
  We analyze benchmark saturation across 60 Large Language Model (LLM) benchmarks selected from technical reports by major model developers.<n>Our analysis reveals that nearly half of the benchmarks exhibit saturation, with rates increasing as benchmarks age.<n>Expert-curated benchmarks resist saturation better than crowdsourced ones.
  arXiv  Detail & Related papers  (2026-02-18T16:51:37Z)
• Benchmark^2: Systematic Evaluation of LLM Benchmarks [66.2731798872668]
  We propose Benchmark2, a comprehensive framework comprising three complementary metrics.<n>We conduct experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains.<n>Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction can achieve comparable evaluation performance.
  arXiv  Detail & Related papers  (2026-01-07T14:59:03Z)
• Adaptive Testing for LLM Evaluation: A Psychometric Alternative to Static Benchmarks [26.89839484242575]
  ATLAS is an adaptive testing framework for large language models.<n>It estimates model ability through Fisher information-guided item selection.<n>It achieves 90% item reduction while maintaining measurement precision.
  arXiv  Detail & Related papers  (2025-10-26T03:54:12Z)
• metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models [5.972993094932516]
  Large Language Models (LLMs) vary in their abilities on a range of tasks.<n>We show that there exists a small set of common underlying abilities that these benchmarks measure.<n>We distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined.
  arXiv  Detail & Related papers  (2024-07-04T17:57:38Z)
• Do Question Answering Modeling Improvements Hold Across Benchmarks? [84.48867898593052]
  We measure concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches. Despite years of intense community focus on a small number of benchmarks, the modeling improvements studied hold broadly.
  arXiv  Detail & Related papers  (2021-02-01T18:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.