LABBench2: An Improved Benchmark for AI Systems Performing Biology Research

• URL: http://arxiv.org/abs/2604.09554v1
• Date: Wed, 04 Feb 2026 18:50:48 GMT
• Title: LABBench2: An Improved Benchmark for AI Systems Performing Biology Research
• Abstract summary: We introduce an evolution of the benchmark for measuring real-world capabilities of AI systems performing scientific tasks.<n>LabBench2 comprises nearly 1,900 tasks and is, for the most part, a continuation of LAB-Bench.<n>We evaluate performance of current frontier models, and show that while abilities measured by LAB-Bench and LABBench2 have improved substantially, LABBench2 provides a meaningful jump in difficulty.
• Score: 33.86687294196014
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Optimism for accelerating scientific discovery with AI continues to grow. Current applications of AI in scientific research range from training dedicated foundation models on scientific data to agentic autonomous hypothesis generation systems to AI-driven autonomous labs. The need to measure progress of AI systems in scientific domains correspondingly must not only accelerate, but increasingly shift focus to more real-world capabilities. Beyond rote knowledge and even just reasoning to actually measuring the ability to perform meaningful work. Prior work introduced the Language Agent Biology Benchmark LAB-Bench as an initial attempt at measuring these abilities. Here we introduce an evolution of that benchmark, LABBench2, for measuring real-world capabilities of AI systems performing useful scientific tasks. LABBench2 comprises nearly 1,900 tasks and is, for the most part, a continuation of LAB-Bench, measuring similar capabilities but in more realistic contexts. We evaluate performance of current frontier models, and show that while abilities measured by LAB-Bench and LABBench2 have improved substantially, LABBench2 provides a meaningful jump in difficulty (model-specific accuracy differences range from -26% to -46% across subtasks) and underscores continued room for performance improvement. LABBench2 continues the legacy of LAB-Bench as a de facto benchmark for AI scientific research capabilities and we hope that it continues to help advance development of AI tools for these core research functions. To facilitate community use and development, we provide the task dataset at https://huggingface.co/datasets/futurehouse/labbench2 and a public eval harness at https://github.com/EdisonScientific/labbench2.

Related papers

• MLS-Bench: A Holistic and Rigorous Assessment of AI Systems on Building Better AI [92.82821272460423]
  MLS-Bench is a benchmark for evaluating whether AI systems can invent generalizable and scalable ML methods.<n>We study the effects of test-time scaling, adaptive compute allocation, and context provision on agents' discovery performance.
  arXiv  Detail & Related papers  (2026-05-09T04:29:46Z)
• AI Co-Scientist for Ranking: Discovering Novel Search Ranking Models alongside LLM-based AI Agents with Cloud Computing Access [60.37646376282544]
  We present an AI Co-Scientist framework that automates the full search ranking research pipeline.<n>This is the first study in the ranking community to utilize an AI Co-Scientist framework for algorithmic research.
  arXiv  Detail & Related papers  (2026-03-23T09:29:08Z)
• Can Large Language Models Derive New Knowledge? A Dynamic Benchmark for Biological Knowledge Discovery [81.03797680309154]
  DBench-Bio is a dynamic and fully automated benchmark to evaluate AI's biological knowledge discovery ability.<n>We instantiate this pipeline to construct a monthly-updated benchmark covering 12 biomedical sub-domains.<n>Our work provides the first dynamic, automatic framework for assessing the new knowledge discovery capabilities of AI systems.
  arXiv  Detail & Related papers  (2026-02-10T05:47:22Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• General Scales Unlock AI Evaluation with Explanatory and Predictive Power [57.7995945974989]
  benchmarking has guided progress in AI, but it has offered limited explanatory and predictive power for general-purpose AI systems.<n>We introduce general scales for AI evaluation that can explain what common AI benchmarks really measure.<n>Our fully-automated methodology builds on 18 newly-crafted rubrics that place instance demands on general scales that do not saturate.
  arXiv  Detail & Related papers  (2025-03-09T01:13:56Z)
• MLGym: A New Framework and Benchmark for Advancing AI Research Agents [51.9387884953294]
  We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing large language models on AI research tasks.<n>This is the first Gym environment for machine learning (ML) tasks, enabling research on reinforcement learning (RL) algorithms for training such agents.<n>We evaluate a number of frontier large language models (LLMs) on our benchmarks such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5 Pro.
  arXiv  Detail & Related papers  (2025-02-20T12:28:23Z)
• CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark [11.794931453828974]
  CORE-Bench is a benchmark consisting of 270 tasks based on 90 scientific papers across three disciplines (computer science, social science, and medicine) We provide an evaluation system to measure the accuracy of agents in a fast and parallelizable way. The best agent achieved an accuracy of 21% on the hardest task, showing the vast scope for improvement in automating routine scientific tasks.
  arXiv  Detail & Related papers  (2024-09-17T17:13:19Z)
• LAB-Bench: Measuring Capabilities of Language Models for Biology Research [1.6312096924271486]
  We introduce the Language Agent Biology Benchmark (LAB-Bench) It is a dataset of over 2,400 multiple choice questions for evaluating AI systems on a range of practical biology research capabilities. We measure performance of several frontier language models against our benchmark and report results compared to human expert biology researchers.
  arXiv  Detail & Related papers  (2024-07-14T23:52:25Z)
• Does AI for science need another ImageNet Or totally different benchmarks? A case study of machine learning force fields [5.622820801789953]
  AI for science (AI4S) aims to enhance the accuracy and speed of scientific computing tasks using machine learning methods. Traditional AI benchmarking methods struggle to adapt to the unique challenges posed by AI4S because they assume data in training, testing, and future real-world queries are independent and identically distributed. This paper investigates the need for a novel approach to effectively benchmark AI for science, using the machine learning force field (MLFF) as a case study.
  arXiv  Detail & Related papers  (2023-08-11T08:06:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.