Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering

• URL: http://arxiv.org/abs/2605.12313v1
• Date: Tue, 12 May 2026 15:59:28 GMT
• Title: Overview of the MedHopQA track at BioCreative IX: track description, participation and evaluation of systems for multi-hop medical question answering
• Abstract summary: The BioCreative IX MedHopQA shared task was designed to benchmark in multi-hop reasoning for large language models (LLMs)<n>We developed a novel dataset of 1,000 challenging QA pairs spanning diseases, genes, and chemicals.<n>Each question was constructed to require two-hop reasoning through the integration of information from two distinct Wikipedia pages.
• Score: 8.26744997684193
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Multi-hop question answering (QA) remains a significant challenge in the biomedical domain, requiring systems to integrate information across multiple sources to answer complex questions. To address this problem, the BioCreative IX MedHopQA shared task was designed to benchmark in multi-hop reasoning for large language models (LLMs). We developed a novel dataset of 1,000 challenging QA pairs spanning diseases, genes, and chemicals, with particular emphasis on rare diseases. Each question was constructed to require two-hop reasoning through the integration of information from two distinct Wikipedia pages. The challenge attracted 48 submissions from 13 teams. Systems were evaluated using both surface string comparison and conceptual accuracy (MedCPT score). The results showed a substantial performance gap between baseline LLMs and enhanced systems. The top-ranked submission achieved an 89.30% F1 score on the MedCPT metric and an 87.30% exact match (EM) score, compared with 67.40% and 60.20%, respectively, for the zero-shot baseline. A central finding of the challenge was that retrieval-augmented generation (RAG) and related retrieval-based strategies were critical for strong performance. In addition, concept-level evaluation improved answer assessment when correct responses differed in surface form. The MedHopQA dataset is publicly available to support continued progress in this important area. Challenge materials: https://www.ncbi.nlm.nih.gov/research/bionlp/medhopqa and benchmark https://www.codabench.org/competitions/7609/

Related papers

• MedHopQA: A Disease-Centered Multi-Hop Reasoning Benchmark and Evaluation Framework for LLM-Based Biomedical Question Answering [11.3883842897598]
  MedHopQA is a disease-centered multi-hop reasoning benchmark consisting of 1,000 expert-curated question-answer pairs.<n>Each question requires synthesis of information across two distinct Wikipedia articles, and answers are provided in an open-ended free-text format.
  arXiv  Detail & Related papers  (2026-05-12T16:32:43Z)
• UETQuintet at BioCreative IX - MedHopQA: Enhancing Biomedical QA with Selective Multi-hop Reasoning and Contextual Retrieval [0.0]
  We propose a model designed to effectively address both direct and sequential questions.<n>We leverage multi-source information retrieval and in-context learning to provide rich, relevant context for generating answers.<n>Our approach achieves an Exact Match score of 0.84, ranking second on the current leaderboard.
  arXiv  Detail & Related papers  (2026-01-11T16:12:38Z)
• CaresAI at BioCreative IX Track 1 -- LLM for Biomedical QA [3.222047196930981]
  Large language models (LLMs) are increasingly evident for accurate question answering across various domains.<n>This paper presents our approach to the MedHopQA track of the BioCreative IX shared task.<n>Three experimental setups are explored: fine-tuning on combined short and long answers, short answers only, and long answers only.
  arXiv  Detail & Related papers  (2025-08-31T11:40:02Z)
• MedQARo: A Large-Scale Benchmark for Medical Question Answering in Romanian [50.767415194856135]
  We introduce MedQARo, the first large-scale medical QA benchmark in Romanian.<n>We construct a high-quality and large-scale dataset comprising 102,646 QA pairs related to cancer patients.
  arXiv  Detail & Related papers  (2025-08-22T13:48:37Z)
• Beyond Retrieval: Ensembling Cross-Encoders and GPT Rerankers with LLMs for Biomedical QA [0.0]
  This paper presents the methodologies and results from our participation in the BioASQ 2025 Task13b Challenge.<n>We built a Retrieval-Augmented Generation (RAG) system that can answer biomedical questions by retrieving relevant PubMed documents and snippets to generate answers.<n>Our solution achieved an MAP@10 of 0.1581, placing 10th on the leaderboard for the retrieval task.
  arXiv  Detail & Related papers  (2025-07-08T01:25:06Z)
• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• Structured Outputs Enable General-Purpose LLMs to be Medical Experts [50.02627258858336]
  Large language models (LLMs) often struggle with open-ended medical questions.<n>We propose a novel approach utilizing structured medical reasoning.<n>Our approach achieves the highest Factuality Score of 85.8, surpassing fine-tuned models.
  arXiv  Detail & Related papers  (2025-03-05T05:24:55Z)
• PeerQA: A Scientific Question Answering Dataset from Peer Reviews [51.95579001315713]
  We present PeerQA, a real-world, scientific, document-level Question Answering dataset.<n>The dataset contains 579 QA pairs from 208 academic articles, with a majority from ML and NLP.<n>We provide a detailed analysis of the collected dataset and conduct experiments establishing baseline systems for all three tasks.
  arXiv  Detail & Related papers  (2025-02-19T12:24:46Z)
• Using Pretrained Large Language Model with Prompt Engineering to Answer Biomedical Questions [1.0742675209112622]
  We propose a two-level information retrieval and question-answering system based on pre-trained large language models (LLM) We construct prompts with in-context few-shot examples and utilize post-processing techniques like resampling and malformed response detection. Our best-performing system achieved 0.14 MAP score on document retrieval, 0.05 MAP score on snippet retrieval, 0.96 F1 score for yes/no questions, 0.38 MRR score for factoid questions and 0.50 F1 score for list questions in Task 12b.
  arXiv  Detail & Related papers  (2024-07-09T11:48:49Z)
• Multi-hop Question Generation with Graph Convolutional Network [58.31752179830959]
  Multi-hop Question Generation (QG) aims to generate answer-related questions by aggregating and reasoning over multiple scattered evidence from different paragraphs. We propose Multi-Hop volution Fusion Network for Question Generation (MulQG), which does context encoding in multiple hops. Our proposed model is able to generate fluent questions with high completeness and outperforms the strongest baseline by 20.8% in the multi-hop evaluation.
  arXiv  Detail & Related papers  (2020-10-19T06:15:36Z)
• Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering [89.76059961309453]
  HeadQA dataset contains multiple-choice questions authorized for the public healthcare specialization exam. These questions are the most challenging for current QA systems. We present a Multi-step reasoning with Knowledge extraction framework (MurKe) We are striving to make full use of off-the-shelf pre-trained models.
  arXiv  Detail & Related papers  (2020-08-06T02:47:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.