BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks

• URL: http://arxiv.org/abs/2608.25286v1
• Date: Wed, 26 Aug 2026 01:45:55 GMT
• Title: BixBench3: Benchmarking AI agents on research-study-scale computational biology tasks
• Abstract summary: We introduce Bix3, a benchmark that measures capacity of AI agents to process biological data through to scientific results.<n>13 models achieve scores ranging from 0.00 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol.<n>On average, agents use 6.8 hours tokens, 102 million, and $43 complete each task, and $43 to the longest attempts consuming 24 hours.
• Score: 1.1546532378724035
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Artificial intelligence (AI) promises to accelerate biological research by automating computational analyses. Yet the ability of AI agents to carry out computational biology at the scale of complete research studies has not been systematically evaluated. Here we introduce BixBench3, a benchmark that measures the capacity of AI agents to process raw biological data through to scientific results. We designed BixBench3 tasks to mirror the delegation of work from a scientist to an agent: the scientist chooses the research question and high-level methods, then delegates implementation of all analyses to the agent. In each task, an agent receives a research objective, methodological guidance, and raw data derived from a published scientific study, and must execute a sequence of analyses to achieve the research objective. The data artifacts resulting from these analyses - such as peak call matrices or differential expression tables - are programmatically graded against the corresponding artifacts generated and reported in the original study. Across 20 BixBench3 tasks encompassing the generation of 138 unique artifacts, we find that 13 frontier models achieve scores ranging from 0.00 for Gemini 3.1 Flash Lite to 0.48 for GPT 5.6 Sol. Agents perform worse on tasks with larger raw datasets (0.36 on tasks with <100 GB versus 0.10 on tasks with >100 GB) and on analyses requiring more sequential steps (0.36 at 1-2 steps vs 0.24 at 3+). On average, agents use 6.8 hours, 102 million tokens, and $43 to complete each task, with the longest attempts consuming 24 hours, 1.07 billion tokens, and $525. Notably, the highest-scoring agents used fewer tokens and were cheaper than less performant options. These results reveal that LLMs vary substantially in their ability to (1) execute multiple sequential analysis steps coherently, (2) manage large quantities of raw data, and (3) work across scientific domains.

Related papers

• Benchmarking AI Agents for Addressing Scientific Challenges Across Scales [118.2204632627895]
  SciAgentArena is a systematic benchmark for evaluating AI agents in real-world scientific research scenarios.<n>It comprises approximately 200 tasks with stepwise verification and an interactive, agent-agnostic environment for assessing diverse AI agents.<n>We find that current agents can contribute effectively to well-specified data-analysis, particularly when the task structure and evaluation criteria are clear.
  arXiv  Detail & Related papers  (2026-06-10T22:55:30Z)
• AIDABench: AI Data Analytics Benchmark [62.45908988324612]
  AIDABench is a benchmark for evaluating AI systems on complex data analytics tasks in an end-to-end manner.<n>AIDABench encompasses 600+ diverse document analysis tasks across three core capability dimensions: question answering, data visualization, and file generation.<n>We evaluate 11 state-of-the-art models on AIDABench, spanning both proprietary (e.g., Claude Sonnet 4.5, Gemini 3 Pro Preview) and open-source (e.g., Qwen3-Max-2026-01-23-Thinking) families.
  arXiv  Detail & Related papers  (2026-02-27T08:58:05Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite [75.58737079136942]
  We present AstaBench, a suite that provides the first holistic measure of agentic ability to perform scientific research.<n>Our suite comes with the first scientific research environment with production-grade search tools.<n>Our evaluation of 57 agents across 22 agent classes reveals several interesting findings.
  arXiv  Detail & Related papers  (2025-10-24T17:10:26Z)
• EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis [0.0]
  Large Language Models (LLMs) offer new opportunities to automate complex interdisciplinary research.<n>EpidemIQs is a novel multi-agent LLM framework that integrates user inputs and autonomously conducts literature review, analytical derivation, network modeling, invoking simulations, data visualization and analysis, and finally documentation of findings in a structured manuscript.<n>We evaluate EpidemIQs across different scenarios measuring computational cost, completion success rate, and AI and human expert reviews of generated reports.
  arXiv  Detail & Related papers  (2025-09-24T18:54:56Z)
• BixBench: a Comprehensive Benchmark for LLM-based Agents in Computational Biology [4.099098082010236]
  Large Language Models (LLMs) and LLM-based agents show great promise in accelerating scientific research.<n>We present the Bioinformatics Benchmark (BixBench), a dataset comprising over 50 real-world scenarios of practical biological data analysis.<n>We evaluate the performance of two frontier LLMs using a custom agent framework we open source.
  arXiv  Detail & Related papers  (2025-02-28T18:47:57Z)
• MLGym: A New Framework and Benchmark for Advancing AI Research Agents [51.9387884953294]
  We introduce Meta MLGym and MLGym-Bench, a new framework and benchmark for evaluating and developing large language models on AI research tasks.<n>This is the first Gym environment for machine learning (ML) tasks, enabling research on reinforcement learning (RL) algorithms for training such agents.<n>We evaluate a number of frontier large language models (LLMs) on our benchmarks such as Claude-3.5-Sonnet, Llama-3.1 405B, GPT-4o, o1-preview, and Gemini-1.5 Pro.
  arXiv  Detail & Related papers  (2025-02-20T12:28:23Z)
• ML Research Benchmark [0.0]
  We present the ML Research Benchmark (MLRB), comprising 7 competition-level tasks derived from recent machine learning conference tracks. This paper introduces a novel benchmark and evaluates it using agent scaffolds powered by frontier models, including Claude-3 and GPT-4o. The results indicate that the Claude-3.5 Sonnet agent performs best across our benchmark, excelling in planning and developing machine learning models.
  arXiv  Detail & Related papers  (2024-10-29T21:38:42Z)
• ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery [23.773528748933934]
  We present ScienceAgentBench, a new benchmark for evaluating language agents for data-driven scientific discovery.<n>We extract 102 tasks from 44 peer-reviewed publications in four disciplines and engage nine subject matter experts to validate them.<n>Using ScienceAgentBench, we evaluate five open-weight and proprietary LLMs, each with three frameworks: direct prompting, OpenHands CodeAct, and self-AI.
  arXiv  Detail & Related papers  (2024-10-07T14:33:50Z)
• DSBench: How Far Are Data Science Agents from Becoming Data Science Experts? [58.330879414174476]
  We introduce DSBench, a benchmark designed to evaluate data science agents with realistic tasks.<n>This benchmark includes 466 data analysis tasks and 74 data modeling tasks, sourced from Eloquence and Kaggle competitions.<n>Our evaluation of state-of-the-art LLMs, LVLMs, and agents shows that they struggle with most tasks, with the best agent solving only 34.12% of data analysis tasks and achieving a 34.74% Relative Performance Gap (RPG)
  arXiv  Detail & Related papers  (2024-09-12T02:08:00Z)
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation [96.71370747681078]
  We introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate.
  arXiv  Detail & Related papers  (2023-10-05T04:06:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.