ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity

• URL: http://arxiv.org/abs/2606.11150v1
• Date: Tue, 09 Jun 2026 17:35:37 GMT
• Title: ABC-Bench: An Agentic Bio-Capabilities Benchmark for Biosecurity
• Abstract summary: Large language models (LLMs) are rapidly acquiring capabilities relevant to biological research.<n>These capabilities offer new opportunities for scientific discovery and biomedical advances, but they also shift the landscape of biosecurity risks.<n>We introduce the Agentic Bio-Capabilities Benchmark (ABC-Bench) to measure agentic biosecurity-relevant capabilities.
• Score: 0.6157382820537719
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) are rapidly acquiring capabilities relevant to biological research, from literature synthesis to interpretation of experimental data. Increasingly, LLM agents can also perform in silico biology tasks that previously required experienced human biologists. These emerging AI capabilities offer new opportunities for scientific discovery and biomedical advances, but they also shift the landscape of biosecurity risks. To address this, we introduce the Agentic Bio-Capabilities Benchmark (ABC-Bench), a suite of tasks to measure agentic biosecurity-relevant capabilities. ABC-Bench evaluates LLM agents on both benign and dual-use biology tasks: writing code to operate liquid handling robots, designing DNA fragments for in vitro assembly, and evading DNA synthesis screening. These tasks require a combination of biology and software expertise. All tested LLM agents outperformed the median expert human baseliner on all three tasks. Agents performed highly on tasks drawing on published knowledge and well-documented protocols, and more weakly on a task requiring novel bioinformatics reasoning. In three wet-lab validation experiments, we found that OpenAI's o4-mini-high produced scripts that, when run on an OpenTrons liquid handling robot, successfully assembled DNA with expected sequences.

Related papers

• Artificial intelligence and biosecurity: capabilities, threat pathways, and defense-in-depth governance [0.0]
  General-purpose large language models can retrieve and integrate scientific information.<n>Biological foundation models can predict, optimize, and generate proteins, genes, and genome-scale sequences.<n> automated laboratories can partially close the design-build-test-learn cycle.
  arXiv  Detail & Related papers  (2026-09-14T18:44:00Z)
• An Early Warning of Emerging Biosecurity Risks in Frontier LLMs [62.01705276160883]
  We develop Intern-BioBreaker, a specialized bio-red-teaming model, together with an integrated computational-to-physical framework.<n>Within this framework, Intern-BioBreaker generates targeted jailbreak prompts to test whether aligned models can be induced to provide operational guidance.<n>Our evaluation reveals a concerning gap between text-level safeguards and the risks posed by capable scientific models.
  arXiv  Detail & Related papers  (2026-07-20T15:26:23Z)
• AIRS-Bench: a Suite of Tasks for Frontier AI Research Science Agents [49.67355440164857]
  We introduce AIRS-Bench, a suite of 20 tasks sourced from state-of-the-art machine learning papers.<n>Airs-Bench tasks assess agentic capabilities over the full research lifecycle.<n>We open-source the AIRS-Bench task definitions and evaluation code to catalyze further development in autonomous scientific research.
  arXiv  Detail & Related papers  (2026-02-06T16:45:02Z)
• BioMARS: A Multi-Agent Robotic System for Autonomous Biological Experiments [8.317138109309967]
  Large language models (LLMs) and vision-language models (VLMs) have the potential to transform biological research by enabling autonomous experimentation.<n>Here we introduce BioMARS, an intelligent platform that integrates LLMs, VLMs, and modular robotics to autonomously design, plan, and execute biological experiments.<n>A web interface enables real-time human-AI collaboration, while a modular backend allows scalable integration with laboratory hardware.
  arXiv  Detail & Related papers  (2025-07-02T08:47:02Z)
• BioProBench: Comprehensive Dataset and Benchmark in Biological Protocol Understanding and Reasoning [31.739027752007928]
  We present BioProBench, the first large-scale, multi-task benchmark for biological protocol understanding and reasoning.<n>Built upon 27K original protocols, it yields nearly 556K high-quality structured instances.
  arXiv  Detail & Related papers  (2025-05-11T09:42:24Z)
• m-KAILIN: Knowledge-Driven Agentic Scientific Corpus Distillation Framework for Biomedical Large Language Models Training [22.996230737442254]
  Corpus Heading for biomedical large language models (LLMs) seeks to address the pressing challenge of insufficient quantity and quality in open-source scientific corpora.<n>This paper proposes a knowledge-driven, agentic framework for scientific corpus distillation, tailored explicitly for LLM training in the biomedical domain.
  arXiv  Detail & Related papers  (2025-04-28T08:18:24Z)
• Biology-Instructions: A Dataset and Benchmark for Multi-Omics Sequence Understanding Capability of Large Language Models [55.74944165932666]
  We introduce Biology-Instructions, the first large-scale instruction-tuning dataset for multi-omics biological sequences.<n>This dataset bridges large language models (LLMs) and complex biological sequence-related tasks, enhancing their versatility and reasoning.<n>We also highlight significant limitations of current state-of-the-art LLMs on multi-omics tasks without specialized training.
  arXiv  Detail & Related papers  (2024-12-26T12:12:23Z)
• Language Model Powered Digital Biology with BRAD [5.309032614374711]
  Large Language Models (LLMs) are well-suited for unstructured integration.<n>We present a prototype Bioinformatics Retrieval Augmented Digital assistant (BRAD)
  arXiv  Detail & Related papers  (2024-09-04T16:43:14Z)
• BioDiscoveryAgent: An AI Agent for Designing Genetic Perturbation Experiments [112.25067497985447]
  We introduce BioDiscoveryAgent, an agent that designs new experiments, reasons about their outcomes, and efficiently navigates the hypothesis space to reach desired solutions.<n>BioDiscoveryAgent can uniquely design new experiments without the need to train a machine learning model.<n>It achieves an average of 21% improvement in predicting relevant genetic perturbations across six datasets.
  arXiv  Detail & Related papers  (2024-05-27T19:57:17Z)
• CRISPR-GPT for Agentic Automation of Gene-editing Experiments [57.10950429181712]
  Large Language Models (LLMs) have shown promise in various tasks, but they often lack specific knowledge and struggle to accurately solve biological design problems.<n>In this work, we introduce CRISPR-GPT, an LLM agent augmented with domain knowledge and external tools to automate and enhance the design process of CRISPR-based gene-editing experiments.<n>We showcase the potential of CRISPR-GPT for assisting non-expert researchers with gene-editing experiments from scratch and validate the agent's effectiveness in a real-world use case.
  arXiv  Detail & Related papers  (2024-04-27T22:59:17Z)
• An Evaluation of Large Language Models in Bioinformatics Research [52.100233156012756]
  We study the performance of large language models (LLMs) on a wide spectrum of crucial bioinformatics tasks. These tasks include the identification of potential coding regions, extraction of named entities for genes and proteins, detection of antimicrobial and anti-cancer peptides, molecular optimization, and resolution of educational bioinformatics problems. Our findings indicate that, given appropriate prompts, LLMs like GPT variants can successfully handle most of these tasks.
  arXiv  Detail & Related papers  (2024-02-21T11:27:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.