Frequency-domain Multi-modal Fusion for Language-guided Medical Image Segmentation

• URL: http://arxiv.org/abs/2509.19719v1
• Date: Wed, 24 Sep 2025 03:02:38 GMT
• Title: Frequency-domain Multi-modal Fusion for Language-guided Medical Image Segmentation
• Abstract summary: The accuracy of medical image segmentation can be improved by incorporating clinical text reports as semantic guidance.<n>We propose a Frequency-domain Multi-modal Interaction model (FMISeg) for language-guided medical image segmentation.
• Score: 15.021321486871658
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatically segmenting infected areas in radiological images is essential for diagnosing pulmonary infectious diseases. Recent studies have demonstrated that the accuracy of the medical image segmentation can be improved by incorporating clinical text reports as semantic guidance. However, the complex morphological changes of lesions and the inherent semantic gap between vision-language modalities prevent existing methods from effectively enhancing the representation of visual features and eliminating semantically irrelevant information, ultimately resulting in suboptimal segmentation performance. To address these problems, we propose a Frequency-domain Multi-modal Interaction model (FMISeg) for language-guided medical image segmentation. FMISeg is a late fusion model that establishes interaction between linguistic features and frequency-domain visual features in the decoder. Specifically, to enhance the visual representation, our method introduces a Frequency-domain Feature Bidirectional Interaction (FFBI) module to effectively fuse frequency-domain features. Furthermore, a Language-guided Frequency-domain Feature Interaction (LFFI) module is incorporated within the decoder to suppress semantically irrelevant visual features under the guidance of linguistic information. Experiments on QaTa-COV19 and MosMedData+ demonstrated that our method outperforms the state-of-the-art methods qualitatively and quantitatively.

Related papers

• Region-Grounded Vision-Language Learning for Detection-Guided Mammographic Lesion Classification [2.9370716053964916]
  We propose a novel region-grounded vision-language learning method for detection-guided mammographic lesion classification.<n>To mitigate semantic collapse and background bias in low-vocabulary settings, we introduce a multi-component objective incorporating positive alignment, fine-grained semantic hard negatives, and background suppression.<n>Experiments on two independent datasets, CBIS-DDSM and VinDr-Mammo, show superior performance of our method compared to related methods under in-domain, cross-dataset, and transfer learning settings.
  arXiv  Detail & Related papers  (2026-07-17T04:27:44Z)
• Fair Cognitive Impairment Detection Through Unlearning [17.94428874004568]
  We present a framework that combines (i) cross-model fusion between modalities (speech, text, and image) and (ii) unlearning using gradient reversal.<n>Our method outperforms the state-of-the-art multilingual and multimodal baseline in MCI classification.<n>We further analyze transfer across datasets, showing that demographic unlearning helps learn more robust representations for MCI detection.
  arXiv  Detail & Related papers  (2026-06-17T00:44:28Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• Plasticine: A Traceable Diffusion Model for Medical Image Translation [79.39689106440389]
  We propose Plasticine, to the best of our knowledge, the first end-to-end image-to-image translation framework explicitly designed with traceability as a core objective.<n>Our method combines intensity translation and spatial transformation within a denoising diffusion framework.<n>This design enables the generation of synthetic images with interpretable intensity transitions and spatially coherent deformations, supporting pixel-wise traceability throughout the translation process.
  arXiv  Detail & Related papers  (2025-12-20T18:01:57Z)
• TCSA-UDA: Text-Driven Cross-Semantic Alignment for Unsupervised Domain Adaptation in Medical Image Segmentation [10.074858409073292]
  We propose a Text-driven Cross-Semantic Alignment framework to guide visual representation learning.<n>Our framework significantly reduces domain shift and consistently outperforms state-of-the-art UDA methods.
  arXiv  Detail & Related papers  (2025-11-08T00:56:52Z)
• Multimodal Causal-Driven Representation Learning for Generalizable Medical Image Segmentation [56.52520416420957]
  We propose Multimodal Causal-Driven Representation Learning (MCDRL) to tackle domain generalization in medical image segmentation.<n>MCDRL consistently outperforms competing methods, yielding superior segmentation accuracy and exhibiting robust generalizability.
  arXiv  Detail & Related papers  (2025-08-07T03:41:41Z)
• Advancing Medical Image Segmentation: Morphology-Driven Learning with Diffusion Transformer [4.672688418357066]
  We propose a novel Transformer Diffusion (DTS) model for robust segmentation in the presence of noise. Our model, which analyzes the morphological representation of images, shows better results than the previous models in various medical imaging modalities.
  arXiv  Detail & Related papers  (2024-08-01T07:35:54Z)
• Enhancing Label-efficient Medical Image Segmentation with Text-guided Diffusion Models [5.865983529245793]
  TextDiff improves semantic representation through inexpensive medical text annotations. We show that TextDiff is significantly superior to the state-of-the-art multi-modal segmentation methods with only a few training samples.
  arXiv  Detail & Related papers  (2024-07-07T10:21:08Z)
• DiffSeg: A Segmentation Model for Skin Lesions Based on Diffusion Difference [2.9082809324784082]
  We introduce DiffSeg, a segmentation model for skin lesions based on diffusion difference. Its multi-output capability mimics doctors' annotation behavior, facilitating the visualization of segmentation result consistency and ambiguity. We demonstrate the effectiveness of DiffSeg on the ISIC 2018 Challenge dataset, outperforming state-of-the-art U-Net-based methods.
  arXiv  Detail & Related papers  (2024-04-25T09:57:52Z)
• FeaInfNet: Diagnosis in Medical Image with Feature-Driven Inference and Visual Explanations [4.022446255159328]
  Interpretable deep learning models have received widespread attention in the field of image recognition. Many interpretability models that have been proposed still have problems of insufficient accuracy and interpretability in medical image disease diagnosis. We propose feature-driven inference network (FeaInfNet) to solve these problems.
  arXiv  Detail & Related papers  (2023-12-04T13:09:00Z)
• Cross-Modal Causal Intervention for Medical Report Generation [107.76649943399168]
  Radiology Report Generation (RRG) is essential for computer-aided diagnosis and medication guidance.<n> generating accurate lesion descriptions remains challenging due to spurious correlations from visual-linguistic biases.<n>We propose a two-stage framework named CrossModal Causal Representation Learning (CMCRL)<n> Experiments on IU-Xray and MIMIC-CXR show that our CMCRL pipeline significantly outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-03-16T07:23:55Z)
• Reliable Joint Segmentation of Retinal Edema Lesions in OCT Images [55.83984261827332]
  In this paper, we propose a novel reliable multi-scale wavelet-enhanced transformer network. We develop a novel segmentation backbone that integrates a wavelet-enhanced feature extractor network and a multi-scale transformer module. Our proposed method achieves better segmentation accuracy with a high degree of reliability as compared to other state-of-the-art segmentation approaches.
  arXiv  Detail & Related papers  (2022-12-01T07:32:56Z)
• Cross-Modality Brain Tumor Segmentation via Bidirectional Global-to-Local Unsupervised Domain Adaptation [61.01704175938995]
  In this paper, we propose a novel Bidirectional Global-to-Local (BiGL) adaptation framework under a UDA scheme. Specifically, a bidirectional image synthesis and segmentation module is proposed to segment the brain tumor. The proposed method outperforms several state-of-the-art unsupervised domain adaptation methods by a large margin.
  arXiv  Detail & Related papers  (2021-05-17T10:11:45Z)
• Few-shot Medical Image Segmentation using a Global Correlation Network with Discriminative Embedding [60.89561661441736]
  We propose a novel method for few-shot medical image segmentation. We construct our few-shot image segmentor using a deep convolutional network trained episodically. We enhance discriminability of deep embedding to encourage clustering of the feature domains of the same class.
  arXiv  Detail & Related papers  (2020-12-10T04:01:07Z)
• Unsupervised Bidirectional Cross-Modality Adaptation via Deeply Synergistic Image and Feature Alignment for Medical Image Segmentation [73.84166499988443]
  We present a novel unsupervised domain adaptation framework, named as Synergistic Image and Feature Alignment (SIFA) Our proposed SIFA conducts synergistic alignment of domains from both image and feature perspectives. Experimental results on two different tasks demonstrate that our SIFA method is effective in improving segmentation performance on unlabeled target images.
  arXiv  Detail & Related papers  (2020-02-06T13:49:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.