Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering

• URL: http://arxiv.org/abs/2608.08307v1
• Date: Sat, 08 Aug 2026 19:36:23 GMT
• Title: Frequency-Domain Dual-Branch Fusion for Medical Visual Question Answering
• Abstract summary: Medical Visual Question Answering (VQA) requires aligning subtle visual evidence, including lesion texture, boundary sharpness, and diffuse density changes, with clinical language.<n>Existing multimodal fusion approaches operating in the spatial domain may not fully exploit complementary frequency information present in visual and textual representations.<n>We introduce a dual-branch frequency-domain fusion module that conditions spectral filtering on the input question, enabling adaptive selection of global low-frequency structure and fine-grained high-frequency detail before reconstructing the spatial representation for answer generation.
• Score: 2.5820404110321413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical Visual Question Answering (VQA) requires aligning subtle visual evidence, including lesion texture, boundary sharpness, and diffuse density changes, with clinical language. Existing multimodal fusion approaches operating in the spatial domain may not fully exploit complementary frequency information present in visual and textual representations. We introduce a dual-branch frequency-domain fusion module that conditions spectral filtering on the input question, enabling adaptive selection of global low-frequency structure and fine-grained high-frequency detail before reconstructing the spatial representation for answer generation. To provide a richer spectrum for filtering, we extract complementary features from early texture-sensitive and final semantic layers of a frozen BiomedCLIP encoder and align both with the question representation using a symmetric InfoNCE objective prior to staged joint training with a BioBART decoder. We pretrain the proposed model on PMC-VQA and fine-tune it on the VQA-RAD and SLAKE benchmarks, demonstrating that frequency-aware multimodal fusion improves medical VQA performance while maintaining a lightweight and efficient architecture.

Related papers

• MedFG-VQA: Low-Frequency Memory and Graph Attention for Lightweight Medical VQA [55.12170695219291]
  MedFG-VQA is a lightweight framework that augments DCT-based low-frequency features and employs graph-enhanced cross-attention for effective visual-textual alignment.<n>To address data scarcity, we construct SynMed-VQA, a large-scale synthetic dataset comprising over 2 million question-answer pairs across 9 imaging modalities and 10 major organs.
  arXiv  Detail & Related papers  (2026-08-27T09:16:14Z)
• RetiWave-Mamba: A Dual-Stream Network for Retinal Disease Detection based on Multi-scale Context and Frequency-Adaptive Mamba Projection [11.498068680313613]
  We propose a novel framework, RetiWave-Mamba, which integrates spatial-frequency domain learning with state-of-the-art state space models.<n>Our approach achieves a state-of-the-art (SOTA) classification accuracy of 98.25%, surpassing existing methods.
  arXiv  Detail & Related papers  (2026-08-18T10:38:59Z)
• HAFI-VLM: A Frequency Perspective for Diagnosing and Enhancing Visual Perception in Vision-Language Models [9.926121668688772]
  Vision-language models (VLMs) remain unreliable when predictions require fine-grained visual evidence.<n>We identify a previously overlooked cause: spectral response rigidity.<n>We propose HAFI-VLM, which introduces a task-conditioned frequency pathway while preserving the pretrained semantic representation.
  arXiv  Detail & Related papers  (2026-08-03T12:15:33Z)
• SonoCLIP: Mask-Guided Region-Aware Vision-Language Pretraining for Fetal Ultrasound Analysis [51.81845420301345]
  SonoCLIP is a vision-language foundation model for fetal ultrasound analysis.<n>It integrates segmentation masks as mask-channel visual prompts within the vision encoder.<n>It achieves superior zero-shot transfer performance under both global and mask-guided inference.
  arXiv  Detail & Related papers  (2026-06-28T20:04:49Z)
• Echo-DM: Ultrasound Marker Removal via Conditional Latent Diffusion and Region-Aware Fusion [50.02433280054874]
  markers can introduce shortcut bias in downstream automated analysis.<n>We present Echo-DM, a framework for ultrasound marker removal via conditional latent diffusion and region-aware fusion.<n>Experiments on Echo-PAIR, a large-scale paired clinical ultrasound dataset, demonstrate superior marker removal and strong anatomical fidelity.
  arXiv  Detail & Related papers  (2026-06-08T11:56:33Z)
• Triple Spectral Fusion for Sensor-based Human Activity Recognition [86.21994396715074]
  This paper proposes a novel triple spectral fusion framework tailored for sensor-based human activity recognition (HAR)<n>We develop an adaptive complementary filtering technique for noise suppression and organize each IMU's sensors into posture and motion modality nodes.<n>An adaptive wavelet frequency selection approach is implemented to suppress context redundancy and shorten the length of features.
  arXiv  Detail & Related papers  (2026-05-04T15:42:58Z)
• WaveSeg: Enhancing Segmentation Precision via High-Frequency Prior and Mamba-Driven Spectrum Decomposition [61.3530659856013]
  We propose a novel decoder architecture, WaveSeg, which jointly optimize feature refinement in spatial and wavelet domains.<n>High-frequency components are first learned from input images as explicit priors to reinforce boundary details.<n>Experiments on standard benchmarks demonstrate that WaveSeg, leveraging wavelet-domain frequency prior with Mamba-based attention, consistently outperforms state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-10-24T01:41:31Z)
• Frequency-Domain Decomposition and Recomposition for Robust Audio-Visual Segmentation [60.9960601057956]
  We introduce Frequency-Aware Audio-Visualcomposer (FAVS) framework consisting of two key modules.<n>FAVS framework achieves state-of-the-art performance on three benchmark datasets.
  arXiv  Detail & Related papers  (2025-09-23T12:33:48Z)
• SpectMamba: Integrating Frequency and State Space Models for Enhanced Medical Image Detection [11.43227481199105]
  We present SpectMamba, the first Mamba-based architecture designed for medical image detection.<n>A key component of SpectMamba is the Hybrid Spatial-Frequency Attention (HSFA) block, which separately learns high- and low-frequency features.<n>We show that SpectMamba achieves state-of-the-art performance while being both effective and efficient across various medical image detection tasks.
  arXiv  Detail & Related papers  (2025-09-01T02:56:45Z)
• Unleashing Vision Foundation Models for Coronary Artery Segmentation: Parallel ViT-CNN Encoding and Variational Fusion [12.839049648094893]
  coronary artery segmentation is critical for computeraided diagnosis of coronary artery disease (CAD)<n>We propose a novel framework that leverages the power of vision foundation models (VFMs) through a parallel encoding architecture.<n>The proposed framework significantly outperforms state-of-the-art methods, achieving superior performance in accurate coronary artery segmentation.
  arXiv  Detail & Related papers  (2025-07-17T09:25:00Z)
• Unified Frequency-Assisted Transformer Framework for Detecting and Grounding Multi-Modal Manipulation [109.1912721224697]
  We present the Unified Frequency-Assisted transFormer framework, named UFAFormer, to address the DGM4 problem. By leveraging the discrete wavelet transform, we decompose images into several frequency sub-bands, capturing rich face forgery artifacts. Our proposed frequency encoder, incorporating intra-band and inter-band self-attentions, explicitly aggregates forgery features within and across diverse sub-bands.
  arXiv  Detail & Related papers  (2023-09-18T11:06:42Z)
• Unlocking Fine-Grained Details with Wavelet-based High-Frequency Enhancement in Transformers [4.208461204572879]
  Medical image segmentation is a critical task that plays a vital role in diagnosis, treatment planning, and disease monitoring. We address the local feature deficiency of the Transformer model by carefully re-designing the self-attention map. We propose a multi-scale context enhancement block within skip connections to adaptively model inter-scale dependencies.
  arXiv  Detail & Related papers  (2023-08-25T15:42:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.