R2MED: A Benchmark for Reasoning-Driven Medical Retrieval

• URL: http://arxiv.org/abs/2505.14558v1
• Date: Tue, 20 May 2025 16:15:30 GMT
• Title: R2MED: A Benchmark for Reasoning-Driven Medical Retrieval
• Authors: Lei Li, Xiao Zhou, Zheng Liu,
• Abstract summary: We introduce R2MED, the first benchmark explicitly designed for reasoning-driven medical retrieval.<n>It comprises 876 queries spanning three tasks: Q&A reference retrieval, clinical evidence retrieval, and clinical case retrieval.<n>We evaluate 15 widely-used retrieval systems on R2MED and find that even the best model achieves only 31.4 nDCG@10.
• Score: 21.743193381874878
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current medical retrieval benchmarks primarily emphasize lexical or shallow semantic similarity, overlooking the reasoning-intensive demands that are central to clinical decision-making. In practice, physicians often retrieve authoritative medical evidence to support diagnostic hypotheses. Such evidence typically aligns with an inferred diagnosis rather than the surface form of a patient's symptoms, leading to low lexical or semantic overlap between queries and relevant documents. To address this gap, we introduce R2MED, the first benchmark explicitly designed for reasoning-driven medical retrieval. It comprises 876 queries spanning three tasks: Q&A reference retrieval, clinical evidence retrieval, and clinical case retrieval. These tasks are drawn from five representative medical scenarios and twelve body systems, capturing the complexity and diversity of real-world medical information needs. We evaluate 15 widely-used retrieval systems on R2MED and find that even the best model achieves only 31.4 nDCG@10, demonstrating the benchmark's difficulty. Classical re-ranking and generation-augmented retrieval methods offer only modest improvements. Although large reasoning models improve performance via intermediate inference generation, the best results still peak at 41.4 nDCG@10. These findings underscore a substantial gap between current retrieval techniques and the reasoning demands of real clinical tasks. We release R2MED as a challenging benchmark to foster the development of next-generation medical retrieval systems with enhanced reasoning capabilities. Data and code are available at https://github.com/R2MED/R2MED

Related papers

• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• Diffusion-driven SpatioTemporal Graph KANsformer for Medical Examination Recommendation [21.649569475134403]
  Recommendation systems in AI-based medical diagnostics and treatment constitute a critical component of AI in healthcare.<n>We first formalize the task of medical examination recommendations.<n>In the first stage, we exploit a task-adaptive diffusion model to distill recommendation-oriented information.<n>In the second stage, a Diffusion graph sformer is proposed to simultaneously model the complex spatial and temporal relationships.
  arXiv  Detail & Related papers  (2025-05-12T10:47:59Z)
• Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QA [17.823588070044217]
  We propose Discuss-RAG, a plug-and-play module designed to enhance the medical question answering system.<n>Our method introduces a summarizer agent that orchestrates a team of medical experts to emulate multi-turn brainstorming, thereby improving the relevance of retrieved content.<n> Experimental results on four benchmark medical QA datasets show that Discuss-RAG consistently outperforms MedRAG.
  arXiv  Detail & Related papers  (2025-04-30T01:37:44Z)
• MedAgentsBench: Benchmarking Thinking Models and Agent Frameworks for Complex Medical Reasoning [34.93995619867384]
  Large Language Models (LLMs) have shown impressive performance on existing medical question-answering benchmarks.<n>We present MedAgentsBench, a benchmark that focuses on challenging medical questions requiring multi-step clinical reasoning, diagnosis formulation, and treatment planning-scenarios.
  arXiv  Detail & Related papers  (2025-03-10T15:38:44Z)
• CliniQ: A Multi-faceted Benchmark for Electronic Health Record Retrieval with Semantic Match Assessment [11.815222175336695]
  We introduce a novel public EHR retrieval benchmark, CliniQ, to address this gap.<n>We build our benchmark upon 1,000 discharge summary notes along with the ICD codes and prescription labels from MIMIC-III.<n>We conduct a comprehensive evaluation of various retrieval methods, ranging from conventional exact match to popular dense retrievers.
  arXiv  Detail & Related papers  (2025-02-10T08:33:47Z)
• MedCoT: Medical Chain of Thought via Hierarchical Expert [48.91966620985221]
  This paper presents MedCoT, a novel hierarchical expert verification reasoning chain method.<n>It is designed to enhance interpretability and accuracy in biomedical imaging inquiries.<n> Experimental evaluations on four standard Med-VQA datasets demonstrate that MedCoT surpasses existing state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-12-18T11:14:02Z)
• Medchain: Bridging the Gap Between LLM Agents and Clinical Practice through Interactive Sequential Benchmarking [58.25862290294702]
  We present MedChain, a dataset of 12,163 clinical cases that covers five key stages of clinical workflow.<n>We also propose MedChain-Agent, an AI system that integrates a feedback mechanism and a MCase-RAG module to learn from previous cases and adapt its responses.
  arXiv  Detail & Related papers  (2024-12-02T15:25:02Z)
• Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering [70.44269982045415]
  Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents.
  arXiv  Detail & Related papers  (2024-11-14T06:19:18Z)
• AutoMIR: Effective Zero-Shot Medical Information Retrieval without Relevance Labels [19.90354530235266]
  We introduce a novel approach called Self-Learning Hypothetical Document Embeddings (SL-HyDE) to tackle this issue. SL-HyDE leverages large language models (LLMs) as generators to generate hypothetical documents based on a given query. We present the Chinese Medical Information Retrieval Benchmark (CMIRB), a comprehensive evaluation framework grounded in real-world medical scenarios.
  arXiv  Detail & Related papers  (2024-10-26T02:53:20Z)
• Semi-Supervised Variational Reasoning for Medical Dialogue Generation [70.838542865384]
  Two key characteristics are relevant for medical dialogue generation: patient states and physician actions. We propose an end-to-end variational reasoning approach to medical dialogue generation. A physician policy network composed of an action-classifier and two reasoning detectors is proposed for augmented reasoning ability.
  arXiv  Detail & Related papers  (2021-05-13T04:14:35Z)
• MedDG: An Entity-Centric Medical Consultation Dataset for Entity-Aware Medical Dialogue Generation [86.38736781043109]
  We build and release a large-scale high-quality Medical Dialogue dataset related to 12 types of common Gastrointestinal diseases named MedDG. We propose two kinds of medical dialogue tasks based on MedDG dataset. One is the next entity prediction and the other is the doctor response generation. Experimental results show that the pre-train language models and other baselines struggle on both tasks with poor performance in our dataset.
  arXiv  Detail & Related papers  (2020-10-15T03:34:33Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.