A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning

• URL: http://arxiv.org/abs/2506.02470v1
• Date: Tue, 03 Jun 2025 05:39:02 GMT
• Title: A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning
• Authors: Xuejiao Zhao, Siyan Liu, Su-Yin Yang, Chunyan Miao,
• Abstract summary: MedRAG is a smart multimodal healthcare copilot equipped with powerful large language model (LLM) reasoning.<n>It supports multiple input modalities, including non-intrusive voice monitoring, general medical queries, and electronic health records.<n>MedRAG retrieves and integrates critical diagnostic insights, reducing the risk of misdiagnosis.
• Score: 47.77948063906033
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Misdiagnosis causes significant harm to healthcare systems worldwide, leading to increased costs and patient risks. MedRAG is a smart multimodal healthcare copilot equipped with powerful large language model (LLM) reasoning, designed to enhance medical decision-making. It supports multiple input modalities, including non-intrusive voice monitoring, general medical queries, and electronic health records. MedRAG provides recommendations on diagnosis, treatment, medication, and follow-up questioning. Leveraging retrieval-augmented generation enhanced by knowledge graph-elicited reasoning, MedRAG retrieves and integrates critical diagnostic insights, reducing the risk of misdiagnosis. It has been evaluated on both public and private datasets, outperforming existing models and offering more specific and accurate healthcare assistance. A demonstration video of MedRAG is available at: https://www.youtube.com/watch?v=PNIBDMYRfDM. The source code is available at: https://github.com/SNOWTEAM2023/MedRAG.

Related papers

• MedGemma Technical Report [75.88152277443179]
  We introduce MedGemma, a collection of medical vision-language foundation models based on Gemma 3 4B and 27B.<n>MedGemma demonstrates advanced medical understanding and reasoning on images and text.<n>We additionally introduce MedSigLIP, a medically-tuned vision encoder derived from SigLIP.
  arXiv  Detail & Related papers  (2025-07-07T17:01:44Z)
• Lingshu: A Generalist Foundation Model for Unified Multimodal Medical Understanding and Reasoning [57.873833577058]
  We build a multimodal dataset enriched with extensive medical knowledge.<n>We then introduce our medical-specialized MLLM: Lingshu.<n>Lingshu undergoes multi-stage training to embed medical expertise and enhance its task-solving capabilities.
  arXiv  Detail & Related papers  (2025-06-08T08:47:30Z)
• Talk Before You Retrieve: Agent-Led Discussions for Better RAG in Medical QA [17.823588070044217]
  We propose Discuss-RAG, a plug-and-play module designed to enhance the medical question answering system.<n>Our method introduces a summarizer agent that orchestrates a team of medical experts to emulate multi-turn brainstorming, thereby improving the relevance of retrieved content.<n> Experimental results on four benchmark medical QA datasets show that Discuss-RAG consistently outperforms MedRAG.
  arXiv  Detail & Related papers  (2025-04-30T01:37:44Z)
• MRD-RAG: Enhancing Medical Diagnosis with Multi-Round Retrieval-Augmented Generation [10.483453944197407]
  We propose a Multi-Round Diagnostic RAG (MRD-RAG) framework that mimics the doctor's diagnostic process.<n>This RAG framework can analyze diagnosis information of potential diseases and accurately conduct multi-round diagnosis like a doctor.
  arXiv  Detail & Related papers  (2025-04-10T13:17:51Z)
• MedRAG: Enhancing Retrieval-augmented Generation with Knowledge Graph-Elicited Reasoning for Healthcare Copilot [47.77948063906033]
  Retrieval-augmented generation (RAG) is a well-suited technique for retrieving privacy-sensitive Electronic Health Records.<n>This paper proposes MedRAG, a RAG model enhanced by knowledge graph (KG)-elicited reasoning for the medical domain.<n>Tests show MedRAG provides more specific diagnostic insights and outperforms state-of-the-art models in reducing misdiagnosis rates.
  arXiv  Detail & Related papers  (2025-02-06T12:27:35Z)
• Medical Multimodal Model Stealing Attacks via Adversarial Domain Alignment [79.41098832007819]
  Medical multimodal large language models (MLLMs) are becoming an instrumental part of healthcare systems.<n>As medical data is scarce and protected by privacy regulations, medical MLLMs represent valuable intellectual property.<n>We introduce Adversarial Domain Alignment (ADA-STEAL), the first stealing attack against medical MLLMs.
  arXiv  Detail & Related papers  (2025-02-04T16:04:48Z)
• MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models [49.765466293296186]
  Recent progress in Medical Large Vision-Language Models (Med-LVLMs) has opened up new possibilities for interactive diagnostic tools.<n>Med-LVLMs often suffer from factual hallucination, which can lead to incorrect diagnoses.<n>We propose a versatile multimodal RAG system, MMed-RAG, designed to enhance the factuality of Med-LVLMs.
  arXiv  Detail & Related papers  (2024-10-16T23:03:27Z)
• MediConfusion: Can you trust your AI radiologist? Probing the reliability of multimodal medical foundation models [20.781551849965357]
  We introduce MediConfusion, a challenging medical Visual Question Answering (VQA) benchmark dataset.<n>We reveal that state-of-the-art models are easily confused by image pairs that are otherwise visually dissimilar and clearly distinct for medical experts.<n>We also extract common patterns of model failure that may help the design of a new generation of more trustworthy and reliable MLLMs in healthcare.
  arXiv  Detail & Related papers  (2024-09-23T18:59:37Z)
• Improving Retrieval-Augmented Generation in Medicine with Iterative Follow-up Questions [42.73799041840482]
  i-MedRAG is a system that iteratively asks follow-up queries based on previous information-seeking attempts. Our zero-shot i-MedRAG outperforms all existing prompt engineering and fine-tuning methods on GPT-3.5. i-MedRAG can flexibly ask follow-up queries to form reasoning chains, providing an in-depth analysis of medical questions.
  arXiv  Detail & Related papers  (2024-08-01T17:18:17Z)
• Capabilities of Gemini Models in Medicine [100.60391771032887]
  We introduce Med-Gemini, a family of highly capable multimodal models specialized in medicine. We evaluate Med-Gemini on 14 medical benchmarks, establishing new state-of-the-art (SoTA) performance on 10 of them. Our results offer compelling evidence for Med-Gemini's potential, although further rigorous evaluation will be crucial before real-world deployment.
  arXiv  Detail & Related papers  (2024-04-29T04:11:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.