RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding

• URL: http://arxiv.org/abs/2605.15561v1
• Date: Fri, 15 May 2026 03:07:52 GMT
• Title: RoiMAM: Region-of-Interest Medical Attention Model for Efficient Vision-Language Understanding
• Abstract summary: Vision-Language Models (VLMs) facilitate medical visual question answering (MedVQA) by jointly interpreting images and text.<n>RoiMAM integrates a training-free ROI Generation Module with Semantic Selective Suppression to focus on lesion-relevant regions.<n>Our design achieves efficient and accurate diagnosis at less than 20% of the model size, while improving accuracy by approximately 2% on SLAKE and 4.6% on PMC-VQA.
• Score: 0.6159852905555453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) facilitate medical visual question answering (MedVQA) by jointly interpreting images and text. However, existing models typically depend on large architectures and closed-set answers, which limits their efficiency and potential clinical applicability. To overcome these shortcomings, we introduce RoiMAM, an efficient VLM. It integrates a training-free ROI Generation Module with Semantic Selective Suppression to focus on lesion-relevant regions, alongside a Text Prompt Enhancer module that provides modality-specific context without introducing training parameters. Compared to the widely used MedVInT-TD model, our design achieves efficient and accurate diagnosis at less than 20\% of the model size, while improving accuracy by approximately 2% on SLAKE and 4.6% on PMC-VQA.

Related papers

• MMBU: A Massive Multi-modal Biomedical Understanding Benchmark to Probe the Perception Capabilities of Vision-Language Models [83.50100003741628]
  Vision and language models (VLMs) hold immense promise to transform biomedical imaging.<n>We introduce the Massive Multimodal Biomedical Understanding (MMBU) benchmark.<n>It is the largest vision and language benchmark to date, covering 35 submodalities with rich structured metadata.
  arXiv  Detail & Related papers  (2026-06-04T20:24:47Z)
• Uncertainty-Aware Vision-Language Segmentation for Medical Imaging [12.545486211087791]
  We introduce a novel uncertainty-aware multimodal segmentation framework for medical diagnosis.<n>We propose a Modality Decoding Attention Block (MoDAB) with a lightweight State Space Mixer (SSMix) to enable efficient cross-modal fusion.<n>Our results highlight the importance of incorporating uncertainty modelling and structured modality alignment in vision-language medical segmentation tasks.
  arXiv  Detail & Related papers  (2026-02-16T06:27:51Z)
• Visual Alignment of Medical Vision-Language Models for Grounded Radiology Report Generation [25.148217482604746]
  We propose VALOR:Visual Alignment of Medical Vision-Language Models for Radiology Report Generation.<n>Our method introduces a reinforcement learning-based post-alignment framework utilizing Group-Relative Proximal Optimization (GRPO)<n>Experiments on multiple benchmarks demonstrate that VALOR substantially improves factual accuracy and visual grounding, achieving significant performance gains over state-of-the-art report generation methods.
  arXiv  Detail & Related papers  (2025-12-18T05:48:21Z)
• Causal-SAM-LLM: Large Language Models as Causal Reasoners for Robust Medical Segmentation [4.286815457787583]
  We introduce Causal-SAM-LLM, a novel framework that elevates Large Language Models (LLMs) to the role of causal reasoners.<n>First, Linguistic Adrial Disentanglement (LAD) employs a Vision-Language Model to generate rich, textual descriptions of confounding image styles.<n>Second, Test-Time Causal Intervention (TCI) provides an interactive mechanism where an LLM interprets a clinician's natural language command to modulate the segmentation decoder's features in real-time.
  arXiv  Detail & Related papers  (2025-07-04T13:52:16Z)
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning [60.03671205298294]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability.<n>This work first proposes a Region-Aware Multimodal Chain-of-Thought dataset, in which the process of producing an answer is preceded by a sequence of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• ExGra-Med: Extended Context Graph Alignment for Medical Vision-Language Models [95.47808515575382]
  ExGra-Med is a novel framework for vision-language integration in medical AI.<n>It aligns images, instruction responses, and extended captions in the latent space, advancing semantic grounding and cross-modal coherence.<n>It matches LLaVA-Med's performance using just 10% of the pre-training data, achieving a 20.13% gain on VQA-RAD and approaching full-data performance.
  arXiv  Detail & Related papers  (2024-10-03T15:52:03Z)
• STLLaVA-Med: Self-Training Large Language and Vision Assistant for Medical Question-Answering [58.79671189792399]
  STLLaVA-Med is designed to train a policy model capable of auto-generating medical visual instruction data. We validate the efficacy and data efficiency of STLLaVA-Med across three major medical Visual Question Answering (VQA) benchmarks.
  arXiv  Detail & Related papers  (2024-06-28T15:01:23Z)
• Fusion of Domain-Adapted Vision and Language Models for Medical Visual Question Answering [4.283761158899643]
  We propose a medical vision-language model that integrates large vision and language models adapted for the medical domain. The proposed model achieves state-of-the-art performance on the SLAKE 1.0 medical VQA dataset with an overall accuracy of 87.5%.
  arXiv  Detail & Related papers  (2024-04-24T20:31:15Z)
• Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluation [113.5002649181103]
  Training open-source small multimodal models (SMMs) to bridge competency gaps for unmet clinical needs in radiology. For training, we assemble a large dataset of over 697 thousand radiology image-text pairs. For evaluation, we propose CheXprompt, a GPT-4-based metric for factuality evaluation, and demonstrate its parity with expert evaluation. The inference of LlaVA-Rad is fast and can be performed on a single V100 GPU in private settings, offering a promising state-of-the-art tool for real-world clinical applications.
  arXiv  Detail & Related papers  (2024-03-12T18:12:02Z)
• Customizing General-Purpose Foundation Models for Medical Report Generation [64.31265734687182]
  The scarcity of labelled medical image-report pairs presents great challenges in the development of deep and large-scale neural networks. We propose customizing off-the-shelf general-purpose large-scale pre-trained models, i.e., foundation models (FMs) in computer vision and natural language processing.
  arXiv  Detail & Related papers  (2023-06-09T03:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.