metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models

• URL: http://arxiv.org/abs/2407.12844v2
• Date: Thu, 20 Feb 2025 16:20:11 GMT
• Title: metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models
• Abstract summary: Large Language Models (LLMs) vary in their abilities on a range of tasks.<n>We show that there exists a small set of common underlying abilities that these benchmarks measure.<n>We distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined.
• Score: 5.972993094932516
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large Language Models (LLMs) vary in their abilities on a range of tasks. Initiatives such as the Open LLM Leaderboard aim to quantify these differences with several large benchmarks (sets of test items to which an LLM can respond either correctly or incorrectly). However, high correlations within and between benchmark scores suggest that (1) there exists a small set of common underlying abilities that these benchmarks measure, and (2) items tap into redundant information and the benchmarks may thus be considerably compressed. We use data from n > 5000 LLMs to identify the most informative items of six benchmarks, ARC, GSM8K, HellaSwag, MMLU, TruthfulQA and WinoGrande (with d = 28,632 items in total). From them we distill a sparse benchmark, metabench, that has less than 3% of the original size of all six benchmarks combined. This new sparse benchmark goes beyond point scores by yielding estimators of the underlying benchmark-specific abilities. We show that these estimators (1) can be used to reconstruct each original individual benchmark score with, on average, 1.24% root mean square error (RMSE), (2) reconstruct the original total score with 0.58% RMSE, and (3) have a single underlying common factor whose Spearman correlation with the total score is r = 0.94.

Related papers

• Zipbench: Low-Cost Framework for Compressing Comprehensive Benchmarks of Large Language Models [27.700132797675007]
  We present ZipBench, a simple and low-cost benchmark suite for large language models (LLMs)<n>ZipBench evaluates only a small set of anchor LLMs, synthesizes pseudo evaluation results to broaden coverage, learns compact sample representations, and selects a small yet representative subset.<n>ZipBench Zoo, a collection of compact versions of 100+ benchmark proxies spanning text, multimodal, and agent tasks, achieves mean absolute errors of 0.002--0.02 and average Spearman correlations of 0.98 with the full benchmarks.
  arXiv  Detail & Related papers  (2026-09-11T06:15:25Z)
• A Statistical Audit of Physical AI Benchmark Redundancy [0.0]
  Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse.<n>We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density.<n>We measure how much information the benchmarks share and show quantitative evidence of Redundancy.
  arXiv  Detail & Related papers  (2026-08-26T15:54:15Z)
• BenchScope: How Many Independent Signals Does Your Benchmark Provide? [0.0]
  We introduce Effective Dimensionality (ED), a fast, population-conditional upper-bound diagnostic of measurement breadth.<n>ED is applied at per-instance granularity to 22 benchmarks across 8 domains and more than 8,400 model evaluations.<n>We show that relative ED rankings are stable under matched-dimension controls and that ED can flag redundant suite components.
  arXiv  Detail & Related papers  (2026-03-31T07:28:41Z)
• Benchmark^2: Systematic Evaluation of LLM Benchmarks [66.2731798872668]
  We propose Benchmark2, a comprehensive framework comprising three complementary metrics.<n>We conduct experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains.<n>Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction can achieve comparable evaluation performance.
  arXiv  Detail & Related papers  (2026-01-07T14:59:03Z)
• Risk Management for Mitigating Benchmark Failure Modes: BenchRisk [7.869587516646358]
  This research iteratively analyzed 26 popular benchmarks, identifying 57 potential failure modes and 196 corresponding mitigation strategies.<n>The mitigations reduce failure likelihood and/or severity, providing a frame for evaluating "benchmark risk"
  arXiv  Detail & Related papers  (2025-10-24T13:43:29Z)
• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• Benchmark Profiling: Mechanistic Diagnosis of LLM Benchmarks [34.09939383415074]
  Benchmark Profiling decomposes benchmark performance into ten cognitively grounded abilities.<n>It explains why performance gains do not always translate into user-perceived competence.
  arXiv  Detail & Related papers  (2025-09-23T15:32:47Z)
• Signal and Noise: A Framework for Reducing Uncertainty in Language Model Evaluation [103.66549325018741]
  We introduce two key metrics that show differences in current benchmarks.<n>We demonstrate that benchmarks with a better signal-to-noise ratio are more reliable when making decisions at small scale.<n>We conclude by recommending that those creating new benchmarks, or selecting which existing benchmarks to use, aim for high signal and low noise.
  arXiv  Detail & Related papers  (2025-08-18T17:56:04Z)
• Importance Sampling is All You Need: Predict LLM's performance on new benchmark by reusing existing benchmark [38.42021928363628]
  Existing benchmarks face two major challenges: (1) the escalating cost of constructing high-quality test suites and reference solutions, and (2) the increasing risk of data contamination.<n>We propose BIS, a prompt-centric evaluation framework that enables ground-truth-free prediction of LLM performance on code generation tasks.<n>Our framework achieves an average absolute prediction error of 1.1% for code correctness scores, with best- and worst-case errors of 0.3% and 1.9%, respectively.
  arXiv  Detail & Related papers  (2025-08-02T05:34:05Z)
• Resampling Benchmark for Efficient Comprehensive Evaluation of Large Vision-Language Models [18.309464845180237]
  We propose an efficient evaluation protocol for large vision-language models (VLMs) We construct a subset that yields results comparable to full benchmark evaluations. Applying FPS to an existing benchmark improves correlation with overall evaluation results.
  arXiv  Detail & Related papers  (2025-04-14T08:43:00Z)
• Information Density Principle for MLLM Benchmarks [59.88484827926759]
  We propose a critical principle of Information Density, which examines how much insight a benchmark can provide for the development of MLLMs. Through a comprehensive analysis of more than 10,000 samples, we measured the information density of 19 MLLM benchmarks. Experiments show that using the latest benchmarks in testing can provide more insight compared to previous ones, but there is still room for improvement in their information density.
  arXiv  Detail & Related papers  (2025-03-13T05:58:41Z)
• Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis [10.133537818749291]
  Large language models (LLMs) have demonstrated significant utilities in real-world applications. Benchmark evaluations are crucial for assessing the capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-13T03:43:33Z)
• How Should We Build A Benchmark? Revisiting 274 Code-Related Benchmarks For LLMs [60.25940747590386]
  We propose How2Bench, which is comprised of a 55-criteria checklist as a set of guidelines to govern the development of code-related benchmarks comprehensively. We profiled 274 benchmarks released within the past decade and found concerning issues. Nearly 70% of the benchmarks did not take measures for data quality assurance; over 10% did not even open source or only partially open source.
  arXiv  Detail & Related papers  (2025-01-18T09:51:57Z)
• MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems [43.19298196163617]
  We develop MIRAGE-Bench, a standardized arena-based multilingual RAG benchmark for 18 diverse languages on Wikipedia. Using this idea, We develop MIRAGE-Bench, a standardized arena-based multilingual RAG benchmark for 18 diverse languages on Wikipedia.
  arXiv  Detail & Related papers  (2024-10-17T16:18:49Z)
• Leaving the barn door open for Clever Hans: Simple features predict LLM benchmark answers [10.786564839628952]
  Internal validity of AI benchmarks is crucial for ensuring they are free from confounding factors. We investigate the possibility that AI systems can solve benchmarks in unintended ways, bypassing the capability being tested.
  arXiv  Detail & Related papers  (2024-10-15T15:05:41Z)
• Data Efficient Evaluation of Large Language Models and Text-to-Image Models via Adaptive Sampling [3.7467864495337624]
  SubLIME is a data-efficient evaluation framework for text-to-image models. Our approach ensures statistically aligned model rankings compared to full datasets. We leverage the HEIM leaderboard to cover 25 text-to-image models on 17 different benchmarks.
  arXiv  Detail & Related papers  (2024-06-21T07:38:55Z)
• Inference-Time Decontamination: Reusing Leaked Benchmarks for Large Language Model Evaluation [61.350306618479365]
  Leakage of benchmarks can prevent the accurate assessment of large language models' true performance. We propose Inference-Time Decontamination (ITD) to address this issue. ITD reduces inflated accuracy by 22.9% on GSM8K and 19.0% on MMLU.
  arXiv  Detail & Related papers  (2024-06-20T04:35:59Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild [57.272096543738336]
  We introduce WildBench, an automated evaluation framework designed to benchmark large language models (LLMs) WildBench consists of 1,024 tasks carefully selected from over one million human-chatbot conversation logs. We have developed two metrics, WB-Reward and WB-Score, which are computable using advanced LLMs.
  arXiv  Detail & Related papers  (2024-06-07T09:15:44Z)
• $\texttt{ACCORD}$: Closing the Commonsense Measurability Gap [16.572584339052753]
  $texttACCORD$ is a framework for disentangling the commonsense grounding and reasoning abilities of large language models. $texttACCORD$ can automatically generate benchmarks of arbitrary reasoning complexity.
  arXiv  Detail & Related papers  (2024-06-04T22:08:24Z)
• MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures [57.886592207948844]
  We propose MixEval, a new paradigm for establishing efficient, gold-standard evaluation by strategically mixing off-the-shelf benchmarks. It bridges (1) comprehensive and well-distributed real-world user queries and (2) efficient and fairly-graded ground-truth-based benchmarks, by matching queries mined from the web with similar queries from existing benchmarks.
  arXiv  Detail & Related papers  (2024-06-03T05:47:05Z)
• How Far Are We on the Decision-Making of LLMs? Evaluating LLMs' Gaming Ability in Multi-Agent Environments [83.78240828340681]
  We introduce GAMA($gamma$)-Bench, a new framework for evaluating Large Language Models' Gaming Ability in Multi-Agent environments. $gamma$-Bench includes eight classical game theory scenarios and a dynamic scoring scheme specially designed to assess LLMs' performance. Results indicate GPT-3.5 demonstrates strong robustness but limited generalizability, which can be enhanced using methods like Chain-of-Thought.
  arXiv  Detail & Related papers  (2024-03-18T14:04:47Z)
• tinyBenchmarks: evaluating LLMs with fewer examples [42.95407654805037]
  We release evaluation tools and tiny versions of popular benchmarks: Open LLM Leaderboard, MMLU, HELM, and AlpacaEval 2.0. Our empirical analysis demonstrates that these tools and tiny benchmarks are sufficient to reliably and efficiently reproduce the original evaluation results.
  arXiv  Detail & Related papers  (2024-02-22T22:05:23Z)
• Linear-time Minimum Bayes Risk Decoding with Reference Aggregation [52.1701152610258]
  Minimum Bayes Risk (MBR) decoding is a text generation technique that has been shown to improve the quality of machine translations. It requires the pairwise calculation of a utility metric, which has quadratic complexity. We propose to approximate pairwise metric scores with scores calculated against aggregated reference representations.
  arXiv  Detail & Related papers  (2024-02-06T18:59:30Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• Likelihood estimation of sparse topic distributions in topic models and its applications to Wasserstein document distance calculations [3.679981089267181]
  In topic models, the $ptimes n$ expected word frequency matrix is assumed to be factorized as a $ptimes K$ word-topic matrix $A$. columns of $A$ are viewed as $p$-dimensional mixture components that are common to all documents. When $A$ is unknown, we estimate $T$ by optimizing the likelihood function corresponding to a plug in, generic, estimator $hatA$ of $A$.
  arXiv  Detail & Related papers  (2021-07-12T22:22:32Z)
• mil-benchmarks: Standardized Evaluation of Deep Multiple-Instance Learning Techniques [0.0]
  Multiple-instance learning is a subset of weakly supervised learning where labels are applied to sets of instances rather than the instances themselves. This paper introduces a series of multiple-instance learning benchmarks generated from MNIST, Fashion-MNIST, and CIFAR10.
  arXiv  Detail & Related papers  (2021-05-04T12:09:55Z)
• Top-$k$ eXtreme Contextual Bandits with Arm Hierarchy [71.17938026619068]
  We study the top-$k$ extreme contextual bandits problem, where the total number of arms can be enormous. We first propose an algorithm for the non-extreme realizable setting, utilizing the Inverse Gap Weighting strategy. We show that our algorithm has a regret guarantee of $O(ksqrt(A-k+1)T log (|mathcalF|T))$.
  arXiv  Detail & Related papers  (2021-02-15T19:10:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.