BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents

• URL: http://arxiv.org/abs/2605.06177v1
• Date: Thu, 07 May 2026 12:57:18 GMT
• Title: BioMedArena: An Open-source Toolkit for Building and Evaluating Biomedical Deep Research Agents
• Abstract summary: We release BioMedArena, an open-source toolkit for evaluating deep-research agents.<n>BioMedArena decouples six layers of biomedical agent evaluation.<n>It exposes 147 biomedical benchmarks and 75 biomedical tools across 9 functional families.
• Score: 35.04967801827194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building a deep research agent today is an exercise in glue code: the same backbone evaluated on the same benchmark can report different accuracies in different papers because harness and tool registry all differ, and integrating a new foundation model into a comparable evaluation surface costs weeks of model-specific engineering. We call this the per-paper engineering tax and release BioMedArena, an open-source toolkit that not only alleviates it but also provides an arena for fair comparison of different foundation models when evaluating them as deep-research agents. BioMedArena decouples six layers of biomedical agent evaluation -- benchmark loading, tool exposure, tool selection, execution mode, context management, and scoring -- and exposes 147 biomedical benchmarks and 75 biomedical tools across 9 functional families. Adding a new model, benchmark, or tool reduces to registering a few-line provider adapter. We further provide 6 agent harnesses with 6 context-management strategies, which provide 12 backbones with competitive research capabilities and significantly improved performance, achieving state-of-the-art (SOTA) results on 8 representative biomedical benchmarks, with an average lift of +15.03 percentage points over prior SOTA. The toolkit, configurations, and per-task traces are available at https://github.com/AI-in-Health/BioMedArena

Related papers

• BioResearcher: Scenario-Guided Multi-Agent for Translational Medicine [1.497336135403194]
  Ingenix BioResearcher is a scenario-guided multi-agent system that maps queries to versioned research playbooks.<n>We evaluate BioResearcher across unit-level capabilities, open-ended biomedical reasoning, and end-to-end clinical discovery.
  arXiv  Detail & Related papers  (2026-05-07T10:33:43Z)
• BioMiner: A Multi-modal System for Automated Mining of Protein-Ligand Bioactivity Data from Literature [53.894504720119805]
  We introduce BioMiner, a multi-modal extraction framework for bioactivity data extraction.<n>Within BioMiner, bioactivity semantics are inferred through direct reasoning, while chemical structures are resolved via a chemical-structure-grounded visual semantic reasoning paradigm.<n>For rigorous evaluation and method development, we establish a benchmark comprising 16,457 bioactivity entries curated from 500 publications.
  arXiv  Detail & Related papers  (2026-04-23T10:11:56Z)
• OpenMed NER: Open-Source, Domain-Adapted State-of-the-Art Transformers for Biomedical NER Across 12 Public Datasets [0.0]
  We introduce OpenMed NER, a suite of open-source, domain-adapted transformer models for named-entity recognition.<n>We evaluate our models on 12 established biomedical NER benchmarks spanning chemicals, diseases, genes, and species.<n>OpenMed NER achieves new state-of-the-art micro-F1 scores on 10 of these 12 datasets, with substantial gains across diverse entity types.
  arXiv  Detail & Related papers  (2025-08-03T07:33:28Z)
• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• Augmenting Biomedical Named Entity Recognition with General-domain Resources [47.24727904076347]
  Training a neural network-based biomedical named entity recognition (BioNER) model usually requires extensive and costly human annotations.<n>We propose GERBERA, a simple-yet-effective method that utilized general-domain NER datasets for training.<n>We systematically evaluated GERBERA on five datasets of eight entity types, collectively consisting of 81,410 instances.
  arXiv  Detail & Related papers  (2024-06-15T15:28:02Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• BiomedGPT: A Generalist Vision-Language Foundation Model for Diverse Biomedical Tasks [68.39821375903591]
  Generalist AI holds the potential to address limitations due to its versatility in interpreting different data types. Here, we propose BiomedGPT, the first open-source and lightweight vision-language foundation model.
  arXiv  Detail & Related papers  (2023-05-26T17:14:43Z)
• BioRED: A Comprehensive Biomedical Relation Extraction Dataset [6.915371362219944]
  We present BioRED, a first-of-its-kind biomedical RE corpus with multiple entity types and relation pairs. We label each relation as describing either a novel finding or previously known background knowledge, enabling automated algorithms to differentiate between novel and background information. Our results show that while existing approaches can reach high performance on the NER task, there is much room for improvement for the RE task.
  arXiv  Detail & Related papers  (2022-04-08T19:23:49Z)
• Benchmarking for Biomedical Natural Language Processing Tasks with a Domain Specific ALBERT [9.8215089151757]
  We present BioALBERT, a domain-specific adaptation of A Lite Bidirectional Representations from Transformers (ALBERT) It is trained on biomedical and PubMed Central and clinical corpora and fine tuned for 6 different tasks across 20 benchmark datasets. It represents a new state of the art in 17 out of 20 benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-09T11:47:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.