Multi-Task Learning for Visually Grounded Reasoning in Gastrointestinal VQA

• URL: http://arxiv.org/abs/2511.04384v1
• Date: Thu, 06 Nov 2025 14:09:56 GMT
• Title: Multi-Task Learning for Visually Grounded Reasoning in Gastrointestinal VQA
• Authors: Itbaan Safwan, Muhammad Annas Shaikh, Muhammad Haaris, Ramail Khan, Muhammad Atif Tahir,
• Abstract summary: We present a multi-task framework for the MediaEval Medico 2025 challenge, leveraging a LoRA-tuned Florence-2 model for simultaneous visual question answering (VQA)<n>The proposed system integrates three curated datasets: (1) Kvasir-VQA-x1 for question-answer learning, (2) a synthetically enriched explanation dataset offering structured medical reasoning, and (3) text-to-region pairs linking visual features with segmentation masks.
• Score: 1.9138416746729587
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a multi-task framework for the MediaEval Medico 2025 challenge, leveraging a LoRA-tuned Florence-2 model for simultaneous visual question answering (VQA), explanation generation, and visual grounding. The proposed system integrates three curated datasets: (1) Kvasir-VQA-x1 for question-answer learning, (2) a synthetically enriched explanation dataset offering structured medical reasoning, and (3) text-to-region pairs linking visual features with segmentation masks. This multi-task setup enables the model to jointly learn visual grounding, reasoning, and interpretation, producing responses that are both accurate and interpretable. Extensive evaluation demonstrates that our approach substantially improves over single-task baselines in both answer accuracy and visual localization, highlighting the effectiveness of grounded multi-task learning for medical VQA applications.

Related papers

• S-Chain: Structured Visual Chain-of-Thought For Medicine [81.97605645734741]
  We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT)<n>The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability.<n>S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical vision-language models.
  arXiv  Detail & Related papers  (2025-10-26T15:57:14Z)
• GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning [60.03671205298294]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability.<n>This work first proposes a Region-Aware Multimodal Chain-of-Thought dataset, in which the process of producing an answer is preceded by a sequence of intermediate reasoning steps.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• Progressive Language-guided Visual Learning for Multi-Task Visual Grounding [21.297317604403652]
  We propose a Progressive Language-guided Visual Learning framework for multi-task visual grounding.<n>In this paper, we propose a Progressive Language-guided Visual Learning framework for multi-task visual grounding.
  arXiv  Detail & Related papers  (2025-04-22T12:48:12Z)
• ViKL: A Mammography Interpretation Framework via Multimodal Aggregation of Visual-knowledge-linguistic Features [54.37042005469384]
  We announce MVKL, the first multimodal mammography dataset encompassing multi-view images, detailed manifestations and reports. Based on this dataset, we focus on the challanging task of unsupervised pretraining. We propose ViKL, a framework that synergizes Visual, Knowledge, and Linguistic features.
  arXiv  Detail & Related papers  (2024-09-24T05:01:23Z)
• Benchmarking Vision-Language Contrastive Methods for Medical Representation Learning [16.03318708001763]
  We perform a comprehensive benchmarking of contrastive frameworks for learning multimodal representations in the medical domain. Our findings suggest a positive answer to the first question, a negative answer to the second question, and the benefit of learning fine-grained features.
  arXiv  Detail & Related papers  (2024-06-11T16:55:38Z)
• Masked Vision and Language Pre-training with Unimodal and Multimodal Contrastive Losses for Medical Visual Question Answering [7.669872220702526]
  We present a novel self-supervised approach that learns unimodal and multimodal feature representations of input images and text. The proposed approach achieves state-of-the-art (SOTA) performance on three publicly available medical VQA datasets.
  arXiv  Detail & Related papers  (2023-07-11T15:00:11Z)
• See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoning [60.43585179885355]
  We propose a novel framework named Interactive Prompting Visual Reasoner (IPVR) for few-shot knowledge-based visual reasoning. IPVR contains three stages, see, think and confirm. We conduct experiments on a range of knowledge-based visual reasoning datasets.
  arXiv  Detail & Related papers  (2023-01-12T18:59:50Z)
• DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention [101.99313208598569]
  Vision-and-language (V-L) tasks require the system to understand both vision content and natural language. We propose DiMBERT (short for Disentangled Multimodal-Attention BERT), which applies separated attention spaces for vision and language. We show that DiMBERT sets new state-of-the-art performance on three tasks.
  arXiv  Detail & Related papers  (2022-10-28T23:00:40Z)
• A Dual-Attention Learning Network with Word and Sentence Embedding for Medical Visual Question Answering [2.0559497209595823]
  Research in medical visual question answering (MVQA) can contribute to the development of computer-aided diagnosis. Existing MVQA question extraction schemes mainly focus on word information, ignoring medical information in the text. In this study, a dual-attention learning network with word and sentence embedding (WSDAN) is proposed.
  arXiv  Detail & Related papers  (2022-10-01T08:32:40Z)
• Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problem [60.0878532426877]
  We propose a novel collaborative learning scheme from the viewpoint of visual perturbation calibration. Specifically, we devise a visual controller to construct two sorts of curated images with different perturbation extents. The experimental results on two diagnostic VQA-CP benchmark datasets evidently demonstrate its effectiveness.
  arXiv  Detail & Related papers  (2022-07-24T23:50:52Z)
• Achieving Human Parity on Visual Question Answering [67.22500027651509]
  The Visual Question Answering (VQA) task utilizes both visual image and language analysis to answer a textual question with respect to an image. This paper describes our recent research of AliceMind-MMU that obtains similar or even slightly better results than human beings does on VQA. This is achieved by systematically improving the VQA pipeline including: (1) pre-training with comprehensive visual and textual feature representation; (2) effective cross-modal interaction with learning to attend; and (3) A novel knowledge mining framework with specialized expert modules for the complex VQA task.
  arXiv  Detail & Related papers  (2021-11-17T04:25:11Z)
• MuVAM: A Multi-View Attention-based Model for Medical Visual Question Answering [2.413694065650786]
  This paper proposes a multi-view attention-based model(MuVAM) for medical visual question answering. It integrates the high-level semantics of medical images on the basis of text description. Experiments on two datasets show that the effectiveness of MuVAM surpasses the state-of-the-art method.
  arXiv  Detail & Related papers  (2021-07-07T13:40:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.