KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering

• URL: http://arxiv.org/abs/2604.00601v1
• Date: Wed, 01 Apr 2026 08:06:37 GMT
• Title: KG-CMI: Knowledge graph enhanced cross-Mamba interaction for medical visual question answering
• Abstract summary: Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine.<n>Recent methods fail to fully leverage domain-specific medical knowledge.<n>Treating Med-VQA as a simple classification problem limits its ability to adapt to the diversity of free-form answers.
• Score: 17.711762075385156
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine. Recent methods fail to fully leverage domain-specific medical knowledge, making it difficult to accurately associate lesion features in medical images with key diagnostic criteria. Additionally, classification-based approaches typically rely on predefined answer sets. Treating Med-VQA as a simple classification problem limits its ability to adapt to the diversity of free-form answers and may overlook detailed semantic information in those answers. To address these challenges, we propose a knowledge graph enhanced cross-Mamba interaction (KG-CMI) framework, which consists of a fine-grained cross-modal feature alignment (FCFA) module, a knowledge graph embedding (KGE) module, a cross-modal interaction representation (CMIR) module, and a free-form answer enhanced multi-task learning (FAMT) module. The KG-CMI learns cross-modal feature representations for images and texts by effectively integrating professional medical knowledge through a graph, establishing associations between lesion features and disease knowledge. Moreover, FAMT leverages auxiliary knowledge from open-ended questions, improving the model's capability for open-ended Med-VQA. Experimental results demonstrate that KG-CMI outperforms existing state-of-the-art methods on three Med-VQA datasets, i.e., VQA-RAD, SLAKE, and OVQA. Additionally, we conduct interpretability experiments to further validate the framework's effectiveness.

Related papers

• CMI-MTL: Cross-Mamba interaction based multi-task learning for medical visual question answering [16.115735955158428]
  Medical visual question answering (Med-VQA) is a crucial multimodal task in clinical decision support and telemedicine.<n>Recent self-attention based methods struggle to handle cross-modal semantic alignments between vision and language.<n>We introduce a Cross-Mamba Interaction based Multi-Task Learning framework that learns cross-modal feature representations from images and texts.
  arXiv  Detail & Related papers  (2025-11-03T09:05:16Z)
• Alignment, Mining and Fusion: Representation Alignment with Hard Negative Mining and Selective Knowledge Fusion for Medical Visual Question Answering [26.129050821950994]
  Medical Visual Question Answering (Med-VQA) is a challenging task that requires a deep understanding of both medical images and textual questions.<n>Our framework outperforms the previous state-of-the-art on widely used Med-VQA datasets like RAD-VQA, SLAKE, PathVQA and VQA 2019.
  arXiv  Detail & Related papers  (2025-10-09T20:14:49Z)
• MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction [1.7775777785480917]
  We propose MedVQA-TREE, a framework that integrates a hierarchical image interpretation module, a gated feature-level fusion mechanism, and a novel multi-hop, multi-query retrieval strategy.<n>A gated fusion mechanism selectively integrates visual features with textual queries, while clinical knowledge is retrieved through a UMLS-guided pipeline accessing PubMed and a sarcopenia-specific external knowledge base.<n>The model achieved up to 99% diagnostic accuracy and outperformed previous state-of-the-art methods by over 10%.
  arXiv  Detail & Related papers  (2025-08-26T13:31:01Z)
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning [60.03671205298294]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability.<n>This work first proposes a Region-Aware Multimodal Chain-of-Thought dataset, in which the process of producing an answer is preceded by a sequence of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• Hierarchical Modeling for Medical Visual Question Answering with Cross-Attention Fusion [4.821565717653691]
  Medical Visual Question Answering (Med-VQA) answers clinical questions using medical images, aiding diagnosis.<n>This study proposes a HiCA-VQA method, including two modules: Hierarchical Prompting for fine-grained medical questions and Hierarchical Answer Decoders.<n> Experiments on the Rad-Restruct benchmark demonstrate that the HiCA-VQA framework better outperforms existing state-of-the-art methods in answering hierarchical fine-grained questions.
  arXiv  Detail & Related papers  (2025-04-04T03:03:12Z)
• ClinKD: Cross-Modal Clinical Knowledge Distiller For Multi-Task Medical Images [4.353855760968461]
  Cross-Modal Clinical Knowledge Distiller (ClinKD) designed to enhance image-text alignment and establish more effective medical knowledge transformation mechanisms.<n>ClinKD achieves state-of-the-art performance on several datasets which are challenging for Med-VQA task.
  arXiv  Detail & Related papers  (2025-02-09T15:08:10Z)
• ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features [54.37042005469384]
  We announce MVKL, the first multimodal mammography dataset encompassing multi-view images, detailed manifestations and reports. Based on this dataset, we focus on the challanging task of unsupervised pretraining. We propose ViKL, a framework that synergizes Visual, Knowledge, and Linguistic features.
  arXiv  Detail & Related papers  (2024-09-24T05:01:23Z)
• FEDMEKI: A Benchmark for Scaling Medical Foundation Models via Federated Knowledge Injection [83.54960238236548]
  FEDMEKI not only preserves data privacy but also enhances the capability of medical foundation models. FEDMEKI allows medical foundation models to learn from a broader spectrum of medical knowledge without direct data exposure.
  arXiv  Detail & Related papers  (2024-08-17T15:18:56Z)
• CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare [16.033112094191395]
  We introduce the Multimodal Medical Question Summarization (MMQS) dataset. This dataset pairs medical queries with visual aids, facilitating a richer and more nuanced understanding of patient needs. We also propose a framework, consisting of four modules that identify medical disorders, generate relevant context, filter medical concepts, and craft visually aware summaries.
  arXiv  Detail & Related papers  (2023-12-16T03:02:05Z)
• PMC-VQA: Visual Instruction Tuning for Medical Visual Question Answering [56.25766322554655]
  Medical Visual Question Answering (MedVQA) presents a significant opportunity to enhance diagnostic accuracy and healthcare delivery. We propose a generative-based model for medical visual understanding by aligning visual information from a pre-trained vision encoder with a large language model. We train the proposed model on PMC-VQA and then fine-tune it on multiple public benchmarks, e.g., VQA-RAD, SLAKE, and Image-Clef 2019.
  arXiv  Detail & Related papers  (2023-05-17T17:50:16Z)
• Multi-task Paired Masking with Alignment Modeling for Medical Vision-Language Pre-training [55.56609500764344]
  We propose a unified framework based on Multi-task Paired Masking with Alignment (MPMA) to integrate the cross-modal alignment task into the joint image-text reconstruction framework. We also introduce a Memory-Augmented Cross-Modal Fusion (MA-CMF) module to fully integrate visual information to assist report reconstruction.
  arXiv  Detail & Related papers  (2023-05-13T13:53:48Z)
• Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining [121.89793208683625]
  Medical artificial general intelligence (MAGI) enables one foundation model to solve different medical tasks. We propose a new paradigm called Medical-knedge-enhanced mulTimOdal pretRaining (MOTOR)
  arXiv  Detail & Related papers  (2023-04-26T01:26:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.