M-Eval: A Heterogeneity-Based Framework for Multi-evidence Validation in Medical RAG Systems

• URL: http://arxiv.org/abs/2510.23995v1
• Date: Tue, 28 Oct 2025 01:57:40 GMT
• Title: M-Eval: A Heterogeneity-Based Framework for Multi-evidence Validation in Medical RAG Systems
• Authors: Mengzhou Sun, Sendong Zhao, Jianyu Chen, Haochun Wang, Bin Qin,
• Abstract summary: Retrieval-augmented Generation (RAG) has demonstrated potential in enhancing medical question-answering systems.<n>This work can help detect errors in current RAG-based medical systems.<n>It also makes the applications of LLMs more reliable and reduces diagnostic errors.
• Score: 21.76710595917909
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented Generation (RAG) has demonstrated potential in enhancing medical question-answering systems through the integration of large language models (LLMs) with external medical literature. LLMs can retrieve relevant medical articles to generate more professional responses efficiently. However, current RAG applications still face problems. They generate incorrect information, such as hallucinations, and they fail to use external knowledge correctly. To solve these issues, we propose a new method named M-Eval. This method is inspired by the heterogeneity analysis approach used in Evidence-Based Medicine (EBM). Our approach can check for factual errors in RAG responses using evidence from multiple sources. First, we extract additional medical literature from external knowledge bases. Then, we retrieve the evidence documents generated by the RAG system. We use heterogeneity analysis to check whether the evidence supports different viewpoints in the response. In addition to verifying the accuracy of the response, we also assess the reliability of the evidence provided by the RAG system. Our method shows an improvement of up to 23.31% accuracy across various LLMs. This work can help detect errors in current RAG-based medical systems. It also makes the applications of LLMs more reliable and reduces diagnostic errors.

Related papers

• MRAG: Benchmarking Retrieval-Augmented Generation for Bio-medicine [3.615835506868351]
  We introduce the Medical Retrieval-Augmented Generation (MRAG) benchmark, covering various tasks in English and Chinese languages.<n>We also develop the MRAG-Toolkit, facilitating systematic exploration of different RAG components.
  arXiv  Detail & Related papers  (2026-01-23T07:07:13Z)
• Optimizing Medical Question-Answering Systems: A Comparative Study of Fine-Tuned and Zero-Shot Large Language Models with RAG Framework [0.0]
  We present a retrieval-augmented generation (RAG) based medical QA system that combines domain-specific knowledge retrieval with open-source LLMs to answer medical questions.<n>We fine-tune two state-of-the-art open LLMs (LLaMA2 and Falcon) using Low-Rank Adaptation (LoRA) for efficient domain specialization.<n>Our fine-tuned LLaMA2 model achieves 71.8% accuracy on PubMedQA, substantially improving over the 55.4% zero-shot baseline.
  arXiv  Detail & Related papers  (2025-12-05T16:38:47Z)
• ReAG: Reasoning-Augmented Generation for Knowledge-based Visual Question Answering [54.72902502486611]
  ReAG is a Reasoning-Augmented Multimodal RAG approach that combines coarse- and fine-grained retrieval with a critic model that filters irrelevant passages.<n>ReAG significantly outperforms prior methods, improving answer accuracy and providing interpretable reasoning grounded in retrieved evidence.
  arXiv  Detail & Related papers  (2025-11-27T19:01:02Z)
• End-to-End Agentic RAG System Training for Traceable Diagnostic Reasoning [52.12425911708585]
  Deep-DxSearch is an agentic RAG system trained end-to-end with reinforcement learning (RL)<n>In Deep-DxSearch, we first construct a large-scale medical retrieval corpus comprising patient records and reliable medical knowledge sources.<n> Experiments demonstrate that our end-to-end RL training framework consistently outperforms prompt-engineering and training-free RAG approaches.
  arXiv  Detail & Related papers  (2025-08-21T17:42:47Z)
• HeteroRAG: A Heterogeneous Retrieval-Augmented Generation Framework for Medical Vision Language Tasks [22.597677744620295]
  We present HeteroRAG, a novel framework that enhances Med-LVLMs through heterogeneous knowledge sources.<n>HeteroRAG achieves state-of-the-art performance in most medical vision language benchmarks.
  arXiv  Detail & Related papers  (2025-08-18T09:54:10Z)
• Faithfulness-Aware Uncertainty Quantification for Fact-Checking the Output of Retrieval Augmented Generation [108.13261761812517]
  We introduce FRANQ (Faithfulness-based Retrieval Augmented UNcertainty Quantification), a novel method for hallucination detection in RAG outputs.<n>We present a new long-form Question Answering (QA) dataset annotated for both factuality and faithfulness.
  arXiv  Detail & Related papers  (2025-05-27T11:56:59Z)
• Fact or Guesswork? Evaluating Large Language Models' Medical Knowledge with Structured One-Hop Judgments [108.55277188617035]
  Large language models (LLMs) have been widely adopted in various downstream task domains, but their abilities to directly recall and apply factual medical knowledge remains under-explored.<n>We introduce the Medical Knowledge Judgment dataset (MKJ), a dataset derived from the Unified Medical Language System (UMLS), a comprehensive repository of standardized vocabularies and knowledge graphs.<n>Through a binary classification framework, MKJ evaluates LLMs' grasp of fundamental medical facts by having them assess the validity of concise, one-hop statements.
  arXiv  Detail & Related papers  (2025-02-20T05:27:51Z)
• Comprehensive and Practical Evaluation of Retrieval-Augmented Generation Systems for Medical Question Answering [70.44269982045415]
  Retrieval-augmented generation (RAG) has emerged as a promising approach to enhance the performance of large language models (LLMs) We introduce Medical Retrieval-Augmented Generation Benchmark (MedRGB) that provides various supplementary elements to four medical QA datasets. Our experimental results reveals current models' limited ability to handle noise and misinformation in the retrieved documents.
  arXiv  Detail & Related papers  (2024-11-14T06:19:18Z)
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models [49.765466293296186]
  Recent progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new possibilities for interactive diagnostic tools.<n>Med-LVLMs often suffer from factual hallucination, which can lead to incorrect diagnoses.<n>We propose a versatile multimodal RAG system, MMed-RAG, designed to enhance the factuality of Med-LVLMs.
  arXiv  Detail & Related papers  (2024-10-16T23:03:27Z)
• Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions [42.73799041840482]
  i-MedRAG is a system that iteratively asks follow-up queries based on previous information-seeking attempts. Our zero-shot i-MedRAG outperforms all existing prompt engineering and fine-tuning methods on GPT-3.5. i-MedRAG can flexibly ask follow-up queries to form reasoning chains, providing an in-depth analysis of medical questions.
  arXiv  Detail & Related papers  (2024-08-01T17:18:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.