BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices

• URL: http://arxiv.org/abs/2411.12990v1
• Date: Wed, 20 Nov 2024 02:38:24 GMT
• Title: BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practices
• Authors: Anka Reuel, Amelia Hardy, Chandler Smith, Max Lamparth, Malcolm Hardy, Mykel J. Kochenderfer,
• Abstract summary: We develop an assessment framework considering 46 best practices across an AI benchmark's lifecycle and evaluate 24 AI benchmarks against it. We find that there exist large quality differences and that commonly used benchmarks suffer from significant issues.
• Score: 28.70453947993952
• License:
• Abstract: AI models are increasingly prevalent in high-stakes environments, necessitating thorough assessment of their capabilities and risks. Benchmarks are popular for measuring these attributes and for comparing model performance, tracking progress, and identifying weaknesses in foundation and non-foundation models. They can inform model selection for downstream tasks and influence policy initiatives. However, not all benchmarks are the same: their quality depends on their design and usability. In this paper, we develop an assessment framework considering 46 best practices across an AI benchmark's lifecycle and evaluate 24 AI benchmarks against it. We find that there exist large quality differences and that commonly used benchmarks suffer from significant issues. We further find that most benchmarks do not report statistical significance of their results nor allow for their results to be easily replicated. To support benchmark developers in aligning with best practices, we provide a checklist for minimum quality assurance based on our assessment. We also develop a living repository of benchmark assessments to support benchmark comparability, accessible at betterbench.stanford.edu.

Related papers

• Improving LLM Leaderboards with Psychometrical Methodology [0.0]
  The rapid development of large language models (LLMs) has necessitated the creation of benchmarks to evaluate their performance. These benchmarks resemble human tests and surveys, as they consist of questions designed to measure emergent properties in the cognitive behavior of these systems. However, unlike the well-defined traits and abilities studied in social sciences, the properties measured by these benchmarks are often vaguer and less rigorously defined.
  arXiv  Detail & Related papers  (2025-01-27T21:21:46Z)
• More than Marketing? On the Information Value of AI Benchmarks for Practitioners [42.73526862595375]
  In academia, public benchmarks were generally viewed as suitable measures for capturing research progress. In product and policy, benchmarks were often found to be inadequate for informing substantive decisions. We conclude that effective benchmarks should provide meaningful, real-world evaluations, incorporate domain expertise, and maintain transparency in scope and goals.
  arXiv  Detail & Related papers  (2024-12-07T03:35:39Z)
• Do These LLM Benchmarks Agree? Fixing Benchmark Evaluation with BenchBench [15.565644819269803]
  We show how some overlooked methodological choices can significantly influence Benchmark Agreement Testing (BAT) results. We introduce BenchBench, a python package for BAT, and release the BenchBench-leaderboard, a meta-benchmark designed to evaluate benchmarks using their peers.
  arXiv  Detail & Related papers  (2024-07-18T17:00:23Z)
• ECBD: Evidence-Centered Benchmark Design for NLP [95.50252564938417]
  We propose Evidence-Centered Benchmark Design (ECBD), a framework which formalizes the benchmark design process into five modules. Each module requires benchmark designers to describe, justify, and support benchmark design choices. Our analysis reveals common trends in benchmark design and documentation that could threaten the validity of benchmarks' measurements.
  arXiv  Detail & Related papers  (2024-06-13T00:59:55Z)
• The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models [94.31327813151208]
  BiGGen Bench is a principled generation benchmark designed to thoroughly evaluate nine distinct capabilities of LMs across 77 diverse tasks. A key feature of the BiGGen Bench is its use of instance-specific evaluation criteria, closely mirroring the nuanced discernment of human evaluation.
  arXiv  Detail & Related papers  (2024-06-09T12:30:30Z)
• Don't Make Your LLM an Evaluation Benchmark Cheater [142.24553056600627]
  Large language models(LLMs) have greatly advanced the frontiers of artificial intelligence, attaining remarkable improvement in model capacity. To assess the model performance, a typical approach is to construct evaluation benchmarks for measuring the ability level of LLMs. We discuss the potential risk and impact of inappropriately using evaluation benchmarks and misleadingly interpreting the evaluation results.
  arXiv  Detail & Related papers  (2023-11-03T14:59:54Z)
• From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
  We discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time. We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Benchmarks for Automated Commonsense Reasoning: A Survey [0.0]
  More than one hundred benchmarks have been developed to test the commonsense knowledge and commonsense reasoning abilities of AI systems. This paper surveys the development and uses of AI commonsense benchmarks.
  arXiv  Detail & Related papers  (2023-02-09T16:34:30Z)
• What Will it Take to Fix Benchmarking in Natural Language Understanding? [30.888416756627155]
  We lay out four criteria that we argue NLU benchmarks should meet. Restoring a healthy evaluation ecosystem will require significant progress in the design of benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-05T20:36:11Z)
• Do Question Answering Modeling Improvements Hold Across Benchmarks? [84.48867898593052]
  We measure concurrence between 32 QA benchmarks on a set of 20 diverse modeling approaches. Despite years of intense community focus on a small number of benchmarks, the modeling improvements studied hold broadly.
  arXiv  Detail & Related papers  (2021-02-01T18:55:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.