Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models

• URL: http://arxiv.org/abs/2606.06534v1
• Date: Wed, 03 Jun 2026 18:12:19 GMT
• Title: Attention Consistent Longitudinal Medical Visual Question Answering Guided by Vision Foundation Models
• Abstract summary: Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point.<n>We propose an attention-guided encoder-decoder for this task with chest X-rays.
• Score: 13.885785530570537
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Longitudinal medical visual question answering (VQA) requires reasoning about anatomical differences between an image of a current time point and an image of a referred time point. We propose an attention-guided encoder-decoder for this task with chest X-rays. Instead of conventional direct contrast, we propose to include a lightweight affine registration module to reduce nuisance motion by co-registering the current image to the reference image with a small registration regularizer. The registered image pair is fed into the image encoder, followed by a frozen DINO-based mask generator and a trainable adaptive mask generator to produce masks applied to the original image pairs. The masked image pairs are again fed into the image encoder and concatenated with text features as the input to a multimodal transformer-based decoder to generate final answers. To facilitate learning stabilization and clarify the change signal, inspired by DINO-v3, we include additional auxiliary objectives, including a mask rebuilding loss, a pairwise Gram-style consistency loss, and a KoLeo uniformity loss, which enhances the geometry of the representation. On the Medical-Diff-VQA benchmark, the model delivers strong BLEU, ROUGE-L, CIDEr, and METEOR scores while offering intrinsic interpretability through the shared saliency mask. These results support saliency-conditioned generation with mild pre-alignment as a principled framework for longitudinal reasoning in medical VQA. Our training strategy also illustrates the potential of a paradigm in utilizing image foundation models in biomedicine: optimizing both supervised and unsupervised learning objectives simultaneously.

Related papers

• Nemotron-Labs-Diffusion-Image: Advancing Masked Discrete Diffusion for High-Resolution Image Synthesis [93.7120829382642]
  Nemotron-Labs-Diffusion-Image is a state-of-the-art masked discrete diffusion model for text-to-image synthesis.<n>Standard MDMs lack self-correcting capability because discrete tokens cannot be modified once they are unmasked.<n>Nemotron-Labs-Diffusion-Image incorporates a token-editing mechanism that enables the model to dynamically revise already-unmasked tokens during inference.
  arXiv  Detail & Related papers  (2026-06-29T05:48:41Z)
• Set-Based Groupwise Registration for Variable-Length, Variable-Contrast Cardiac MRI [5.342753447867402]
  Groupwise registration has shown greater robustness than pairwise registration for motion correction.<n>emphAnyTwoReg takes a quantitative MRI sequence as an unordered set.<n>It achieves strong cross-protocol generalization in a zero-shot manner.
  arXiv  Detail & Related papers  (2026-05-11T13:41:22Z)
• VQRAE: Representation Quantization Autoencoders for Multimodal Understanding, Generation and Reconstruction [83.50898344094153]
  VQRAE produces Continuous semantic features for image understanding and tokens for visual generation within a unified tokenizer.<n>Design enables negligible semantic information for maintaining the ability of multimodal understanding, discrete tokens.<n>VQRAE presents competitive performance on several benchmarks of visual understanding, generation and reconstruction.
  arXiv  Detail & Related papers  (2025-11-28T17:26:34Z)
• Saliency Guided Longitudinal Medical Visual Question Answering [9.991158264639209]
  Longitudinal medical visual question answering (Diff-VQA) requires comparing paired studies from different time points.<n>We propose a saliency-guided encoder-decoder for chest X-ray Diff-VQA that turns post-hoc saliency into actionable supervision.
  arXiv  Detail & Related papers  (2025-09-29T18:26:17Z)
• SimGen: A Diffusion-Based Framework for Simultaneous Surgical Image and Segmentation Mask Generation [1.9393128408121891]
  generative AI models like text-to-image can alleviate data scarcity, incorporating spatial annotations, such as segmentation masks, is crucial for precision-driven surgical applications, simulation, and education.<n>This study introduces both a novel task and method, SimGen, for Simultaneous Image and Mask Generation.<n>SimGen is a diffusion model based on the DDPM framework and Residual U-Net, designed to jointly generate high-fidelity surgical images and their corresponding segmentation masks.
  arXiv  Detail & Related papers  (2025-01-15T18:48:38Z)
• Neural Image Compression Using Masked Sparse Visual Representation [17.229601298529825]
  We study neural image compression based on the Sparse Visual Representation (SVR), where images are embedded into a discrete latent space spanned by learned visual codebooks. By sharing codebooks with the decoder, the encoder transfers codeword indices that are efficient and cross-platform robust. We propose a Masked Adaptive Codebook learning (M-AdaCode) method that applies masks to the latent feature subspace to balance and reconstruction quality.
  arXiv  Detail & Related papers  (2023-09-20T21:59:23Z)
• Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training [51.16994853817024]
  This work focuses on designing an effective pre-training framework for 3D radiology images. We introduce Disruptive Autoencoders, a pre-training framework that attempts to reconstruct the original image from disruptions created by a combination of local masking and low-level perturbations. The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-31T17:59:42Z)
• Not All Image Regions Matter: Masked Vector Quantization for Autoregressive Image Generation [78.13793505707952]
  Existing autoregressive models follow the two-stage generation paradigm that first learns a codebook in the latent space for image reconstruction and then completes the image generation autoregressively based on the learned codebook. We propose a novel two-stage framework, which consists of Masked Quantization VAE (MQ-VAE) Stack model from modeling redundancy.
  arXiv  Detail & Related papers  (2023-05-23T02:15:53Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering [7.2486693553383805]
  Current Medical-VQA models learn cross-modal representations through residing vision and texture encoders in dual separate spaces. We propose UnICLAM, a Unified and Interpretable Medical-VQA model through Contrastive Representation Learning with Adversarial Masking. Experimental results on VQA-RAD and SLAKE public benchmarks demonstrate that UnICLAM outperforms existing 11 state-of-the-art Medical-VQA models.
  arXiv  Detail & Related papers  (2022-12-21T02:48:15Z)
• MAGIC: Mask-Guided Image Synthesis by Inverting a Quasi-Robust Classifier [37.774220727662914]
  We propose a one-shot mask-guided image synthesis that allows controlling manipulations of a single image. Our proposed method, entitled MAGIC, leverages structured gradients from a pre-trained quasi-robust classifier. MAGIC aggregates gradients over the input, driven by a guide binary mask that enforces a strong, spatial prior.
  arXiv  Detail & Related papers  (2022-09-23T12:15:40Z)
• Attentive Symmetric Autoencoder for Brain MRI Segmentation [56.02577247523737]
  We propose a novel Attentive Symmetric Auto-encoder based on Vision Transformer (ViT) for 3D brain MRI segmentation tasks. In the pre-training stage, the proposed auto-encoder pays more attention to reconstruct the informative patches according to the gradient metrics. Experimental results show that our proposed attentive symmetric auto-encoder outperforms the state-of-the-art self-supervised learning methods and medical image segmentation models.
  arXiv  Detail & Related papers  (2022-09-19T09:43:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.