BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research

• URL: http://arxiv.org/abs/2505.16100v1
• Date: Thu, 22 May 2025 01:02:21 GMT
• Title: BioDSA-1K: Benchmarking Data Science Agents for Biomedical Research
• Authors: Zifeng Wang, Benjamin Danek, Jimeng Sun,
• Abstract summary: BioDSA-1K consists of 1,029 hypothesis-centric tasks paired with 1,177 analysis plans.<n>The benchmark enables evaluation along four axes: (1) hypothesis decision accuracy, (2) alignment between evidence and conclusion, (3) correctness of the reasoning process, and (4) executability of the AI-generated analysis code.
• Score: 29.469867701731374
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Validating scientific hypotheses is a central challenge in biomedical research, and remains difficult for artificial intelligence (AI) agents due to the complexity of real-world data analysis and evidence interpretation. In this work, we present BioDSA-1K, a benchmark designed to evaluate AI agents on realistic, data-driven biomedical hypothesis validation tasks. BioDSA-1K consists of 1,029 hypothesis-centric tasks paired with 1,177 analysis plans, curated from over 300 published biomedical studies to reflect the structure and reasoning found in authentic research workflows. Each task includes a structured hypothesis derived from the original study's conclusions, expressed in the affirmative to reflect the language of scientific reporting, and one or more pieces of supporting evidence grounded in empirical data tables. While these hypotheses mirror published claims, they remain testable using standard statistical or machine learning methods. The benchmark enables evaluation along four axes: (1) hypothesis decision accuracy, (2) alignment between evidence and conclusion, (3) correctness of the reasoning process, and (4) executability of the AI-generated analysis code. Importantly, BioDSA-1K includes non-verifiable hypotheses: cases where the available data are insufficient to support or refute a claim, reflecting a common yet underexplored scenario in real-world science. We propose BioDSA-1K as a foundation for building and evaluating generalizable, trustworthy AI agents for biomedical discovery.

Related papers

• Automating Exploratory Multiomics Research via Language Models [22.302672656499315]
  PROTEUS is a fully automated system that produces data-driven hypotheses from raw data files.<n>We apply PROTEUS to clinical proteogenomics, a field where effective downstream data analysis and hypothesis proposal is crucial for producing novel discoveries.
  arXiv  Detail & Related papers  (2025-06-09T09:44:21Z)
• Benchmarking AI scientists in omics data-driven biological research [3.3605177939410713]
  We introduce the Biological AI Scientist Benchmark (BaisBench) to assess AI scientists' ability to generate biological discoveries.<n>BaisBench comprises two tasks: cell type annotation on 31 expert-labeled single-cell datasets, and scientific discovery through answering 198 multiple-choice questions.
  arXiv  Detail & Related papers  (2025-05-13T08:33:54Z)
• Towards an AI co-scientist [48.11351101913404]
  We introduce an AI co-scientist, a multi-agent system built on Gemini 2.0.<n>The AI co-scientist is intended to help uncover new, original knowledge and to formulate demonstrably novel research hypotheses.<n>The system's design incorporates a generate, debate, and evolve approach to hypothesis generation, inspired by the scientific method.
  arXiv  Detail & Related papers  (2025-02-26T06:17:13Z)
• Causal Representation Learning from Multimodal Biomedical Observations [57.00712157758845]
  We develop flexible identification conditions for multimodal data and principled methods to facilitate the understanding of biomedical datasets.<n>Key theoretical contribution is the structural sparsity of causal connections between modalities.<n>Results on a real-world human phenotype dataset are consistent with established biomedical research.
  arXiv  Detail & Related papers  (2024-11-10T16:40:27Z)
• Automating Exploratory Proteomics Research via Language Models [22.302672656499315]
  PROTEUS is a fully automated system for scientific discovery from raw data. It produces a comprehensive set of research objectives, analysis results and novel biological hypotheses without human intervention.
  arXiv  Detail & Related papers  (2024-11-06T08:16:56Z)
• Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation [15.495976478018264]
  Large language models (LLMs) have emerged as a promising tool to revolutionize knowledge interaction. We construct a dataset of background-hypothesis pairs from biomedical literature, partitioned into training, seen, and unseen test sets. We assess the hypothesis generation capabilities of top-tier instructed models in zero-shot, few-shot, and fine-tuning settings.
  arXiv  Detail & Related papers  (2024-07-12T02:55:13Z)
• BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments [112.25067497985447]
  We introduce BioDiscoveryAgent, an agent that designs new experiments, reasons about their outcomes, and efficiently navigates the hypothesis space to reach desired solutions.<n>BioDiscoveryAgent can uniquely design new experiments without the need to train a machine learning model.<n>It achieves an average of 21% improvement in predicting relevant genetic perturbations across six datasets.
  arXiv  Detail & Related papers  (2024-05-27T19:57:17Z)
• Large Language Models for Automated Open-domain Scientific Hypotheses Discovery [50.40483334131271]
  This work proposes the first dataset for social science academic hypotheses discovery. Unlike previous settings, the new dataset requires (1) using open-domain data (raw web corpus) as observations; and (2) proposing hypotheses even new to humanity. A multi- module framework is developed for the task, including three different feedback mechanisms to boost performance.
  arXiv  Detail & Related papers  (2023-09-06T05:19:41Z)
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)
• CBLUE: A Chinese Biomedical Language Understanding Evaluation Benchmark [51.38557174322772]
  We present the first Chinese Biomedical Language Understanding Evaluation benchmark. It is a collection of natural language understanding tasks including named entity recognition, information extraction, clinical diagnosis normalization, single-sentence/sentence-pair classification. We report empirical results with the current 11 pre-trained Chinese models, and experimental results show that state-of-the-art neural models perform by far worse than the human ceiling.
  arXiv  Detail & Related papers  (2021-06-15T12:25:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.