Saliency Guided Longitudinal Medical Visual Question Answering

• URL: http://arxiv.org/abs/2509.25374v1
• Date: Mon, 29 Sep 2025 18:26:17 GMT
• Title: Saliency Guided Longitudinal Medical Visual Question Answering
• Abstract summary: Longitudinal medical visual question answering (Diff-VQA) requires comparing paired studies from different time points.<n>We propose a saliency-guided encoder-decoder for chest X-ray Diff-VQA that turns post-hoc saliency into actionable supervision.
• Score: 9.991158264639209
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Longitudinal medical visual question answering (Diff-VQA) requires comparing paired studies from different time points and answering questions about clinically meaningful changes. In this setting, the difference signal and the consistency of visual focus across time are more informative than absolute single-image findings. We propose a saliency-guided encoder-decoder for chest X-ray Diff-VQA that turns post-hoc saliency into actionable supervision. The model first performs a lightweight near-identity affine pre-alignment to reduce nuisance motion between visits. It then executes a within-epoch two-step loop: step 1 extracts a medically relevant keyword from the answer and generates keyword-conditioned Grad-CAM on both images to obtain disease-focused saliency; step 2 applies the shared saliency mask to both time points and generates the final answer. This closes the language-vision loop so that the terms that matter also guide where the model looks, enforcing spatially consistent attention on corresponding anatomy. On Medical-Diff-VQA, the approach attains competitive performance on BLEU, ROUGE-L, CIDEr, and METEOR while providing intrinsic interpretability. Notably, the backbone and decoder are general-domain pretrained without radiology-specific pretraining, highlighting practicality and transferability. These results support saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning in medical VQA.

Related papers

• RASA: Disentangled Spatial-Motional Priors for Cross-Identity Character Animation [85.41886214825894]
  Cross-identity character animation aims to drive a target identity from a reference image to follow the motion of a source character from a driving video.<n>We introduce Reference-Aware Structural Alignment (RASA), a framework that disentangles spatial mapping from motion control.<n>Our work establishes a new paradigm showing disentangled spatial and motional priors are key to robust character animation.
  arXiv  Detail & Related papers  (2026-08-28T11:36:44Z)
• Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering [2.5820404110321413]
  Medical Visual Question Answering (VQA) requires aligning subtle visual evidence, including lesion texture, boundary sharpness, and diffuse density changes, with clinical language.<n>Existing multimodal fusion approaches operating in the spatial domain may not fully exploit complementary frequency information present in visual and textual representations.<n>We introduce a dual-branch frequency-domain fusion module that conditions spectral filtering on the input question, enabling adaptive selection of global low-frequency structure and fine-grained high-frequency detail before reconstructing the spatial representation for answer generation.
  arXiv  Detail & Related papers  (2026-08-08T19:36:23Z)
• LoMeVQA: A Comprehensive Benchmark for Longitudinal Medical VQA [23.762697050770157]
  We propose LoMeVQA, a benchmark consisting of 206K longitudinal visual question answering (VQA) pairs for temporal medical image analysis.<n>LoMeVQA covers five tasks: progress classification, progress description, progress report generation, differential region grounding, and differential region description.<n>We introduce MedLong-8B, which achieves state-of-the-art performance across all tasks.
  arXiv  Detail & Related papers  (2026-07-30T07:45:58Z)
• See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs [81.71384150441163]
  We propose Context-aware Attention Intervention (CAI) for large vision-language models (LVLMs)<n>CAI enforces a see only when needed principle via two-axis selectivity: where to look and when to intervene.<n> experiments across multiple LVLM backbones and benchmarks show that CAI achieves state-of-the-art hallucination mitigation.
  arXiv  Detail & Related papers  (2026-06-29T06:35:47Z)
• Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models [13.885785530570537]
  Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point.<n>We propose an attention-guided encoder-decoder for this task with chest X-rays.
  arXiv  Detail & Related papers  (2026-06-03T18:12:19Z)
• V-Loop: Visual Logical Loop Verification for Hallucination Detection in Medical Visual Question Answering [8.994256499686863]
  We propose a training-free and plug-and-play framework for hallucination detection in medical visual question answering.<n>V-Loop forms a visually grounded logical loop to verify factual correctness.<n>It consistently outperforms existing introspective methods, remains highly efficient, and further boosts uncertainty-based approaches when used in combination.
  arXiv  Detail & Related papers  (2026-01-26T07:46:41Z)
• See Less, See Right: Bi-directional Perceptual Shaping For Multimodal Reasoning [58.7125460363147]
  We propose Bi-directional Perceptual Shaping (BiPS), which transforms question-conditioned masked views into bidirectional where-to-look signals.<n>BiPS boosts Qwen2.5-VL-7B by 8.2% on average and shows strong out-of-domain generalization to unseen datasets and image types.
  arXiv  Detail & Related papers  (2025-12-26T18:59:47Z)
• SurgViVQA: Temporally-Grounded Video Question Answering for Surgical Scene Understanding [11.424693319143715]
  Video Question Answering (VideoQA) in the surgical domain aims to enhance intraoperative understanding by enabling AI models to reason over temporally coherent events.<n>We propose SurgViVQA, a model that extends visual reasoning from static images to dynamic surgical scenes.<n>It uses a Masked Video-Text to fuse video and question features, capturing temporal cues such as motion and tool--tissue interactions.
  arXiv  Detail & Related papers  (2025-11-05T09:40:16Z)
• SurgAnt-ViVQA: Learning to Anticipate Surgical Events through GRU-Driven Temporal Cross-Attention [10.149538951173598]
  Anticipating forthcoming surgical events is vital for real-time assistance in endonasal transsphenoidal pituitary surgery.<n>Most visual question answering (VQA) systems reason on isolated frames with static vision language alignment.<n>We introduce PitVQA-Anticipation, the first VQA dataset designed for forward looking surgical reasoning.
  arXiv  Detail & Related papers  (2025-11-05T04:55:11Z)
• S-Chain: Structured Visual Chain-of-Thought For Medicine [81.97605645734741]
  We introduce S-Chain, the first large-scale dataset of 12,000 expert-annotated medical images with bounding boxes and structured visual CoT (SV-CoT)<n>The dataset further supports 16 languages, totaling over 700k VQA pairs for broad multilingual applicability.<n>S-Chain establishes a new benchmark for grounded medical reasoning and paves the way toward more trustworthy and explainable medical vision-language models.
  arXiv  Detail & Related papers  (2025-10-26T15:57:14Z)
• CardiacCLIP: Video-based CLIP Adaptation for LVEF Prediction in a Few-shot Manner [14.429336783145644]
  Left ventricular ejection fraction (LVEF) serves as a key indicator of heart function.<n>Existing LVEF estimation methods depend on large-scale annotated video datasets.<n>We propose CardiacCLIP, a video-based framework that enhances LVEF prediction through attention-based frame aggregation and multi-resolution input scaling.
  arXiv  Detail & Related papers  (2025-09-21T12:52:08Z)
• GEMeX-ThinkVG: Towards Thinking with Visual Grounding in Medical VQA via Reinforcement Learning [50.94508930739623]
  Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images.<n>Current methods still suffer from limited answer reliability and poor interpretability, impairing the ability of clinicians and patients to understand and trust model-generated answers.<n>This work first proposes a Thinking with Visual Grounding dataset wherein the answer generation is decomposed into intermediate reasoning steps.<n>We introduce a novel verifiable reward mechanism for reinforcement learning to guide post-training, improving the alignment between the model's reasoning process and its final answer.
  arXiv  Detail & Related papers  (2025-06-22T08:09:58Z)
• Seeing the Trees for the Forest: Rethinking Weakly-Supervised Medical Visual Grounding [45.248265283405004]
  Current models struggle to associate textual descriptions with disease regions due to inefficient attention mechanisms and a lack of fine-grained token representations.<n>We introduce Disease-Aware Prompting (DAP), which uses the explainability map of a VLM to identify the appropriate image features.<n>DAP improves visual grounding accuracy by 20.74% compared to state-of-the-art methods across three major chest X-ray datasets.
  arXiv  Detail & Related papers  (2025-05-21T05:16:45Z)
• Longitudinal Segmentation of MS Lesions via Temporal Difference Weighting [2.0168790328644697]
  We introduce a novel approach that explicitly incorporates temporal differences between baseline and follow-up scans through a unique architectural inductive bias called Difference Weighting Block. We achieve superior scores in lesion segmentation as well as lesion detection as compared to state-of-the-art longitudinal and single timepoint models across two datasets.
  arXiv  Detail & Related papers  (2024-09-20T11:30:54Z)
• Real-time guidewire tracking and segmentation in intraoperative x-ray [52.51797358201872]
  We propose a two-stage deep learning framework for real-time guidewire segmentation and tracking. In the first stage, a Yolov5 detector is trained, using the original X-ray images as well as synthetic ones, to output the bounding boxes of possible target guidewires. In the second stage, a novel and efficient network is proposed to segment the guidewire in each detected bounding box.
  arXiv  Detail & Related papers  (2024-04-12T20:39:19Z)
• Learning to Exploit Temporal Structure for Biomedical Vision-Language Processing [53.89917396428747]
  Self-supervised learning in vision-language processing exploits semantic alignment between imaging and text modalities. We explicitly account for prior images and reports when available during both training and fine-tuning. Our approach, named BioViL-T, uses a CNN-Transformer hybrid multi-image encoder trained jointly with a text model.
  arXiv  Detail & Related papers  (2023-01-11T16:35:33Z)
• UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering [7.2486693553383805]
  Current Medical-VQA models learn cross-modal representations through residing vision and texture encoders in dual separate spaces. We propose UnICLAM, a Unified and Interpretable Medical-VQA model through Contrastive Representation Learning with Adversarial Masking. Experimental results on VQA-RAD and SLAKE public benchmarks demonstrate that UnICLAM outperforms existing 11 state-of-the-art Medical-VQA models.
  arXiv  Detail & Related papers  (2022-12-21T02:48:15Z)
• Cloth-Changing Person Re-identification from A Single Image with Gait Prediction and Regularization [65.50321170655225]
  We introduce Gait recognition as an auxiliary task to drive the Image ReID model to learn cloth-agnostic representations. Experiments on image-based Cloth-Changing ReID benchmarks, e.g., LTCC, PRCC, Real28, and VC-Clothes, demonstrate that GI-ReID performs favorably against the state-of-the-arts.
  arXiv  Detail & Related papers  (2021-03-29T12:10:50Z)
• BiteNet: Bidirectional Temporal Encoder Network to Predict Medical Outcomes [53.163089893876645]
  We propose a novel self-attention mechanism that captures the contextual dependency and temporal relationships within a patient's healthcare journey. An end-to-end bidirectional temporal encoder network (BiteNet) then learns representations of the patient's journeys. We have evaluated the effectiveness of our methods on two supervised prediction and two unsupervised clustering tasks with a real-world EHR dataset.
  arXiv  Detail & Related papers  (2020-09-24T00:42:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.