Test-Time Modality Generalization for Medical Image Segmentation

• URL: http://arxiv.org/abs/2502.19671v1
• Date: Thu, 27 Feb 2025 01:32:13 GMT
• Title: Test-Time Modality Generalization for Medical Image Segmentation
• Authors: Ju-Hyeon Nam, Sang-Chul Lee,
• Abstract summary: Generalizable medical image segmentation is essential for ensuring consistent performance across diverse unseen clinical settings.<n>We introduce a novel Test-Time Modality Generalization (TTMG) framework, which comprises two core components: Modality-Aware Style Projection (MASP) and Modality-Sensitive Instance Whitening (MSIW)<n>MASP estimates the likelihood of a test instance belonging to each seen modality and maps it onto a distribution using modality-specific style bases, guiding its projection effectively.<n>MSIW is applied during training to selectively suppress modality-sensitive information while retaining modality-invariant features.
• Score: 0.9092907230570326
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Generalizable medical image segmentation is essential for ensuring consistent performance across diverse unseen clinical settings. However, existing methods often overlook the capability to generalize effectively across arbitrary unseen modalities. In this paper, we introduce a novel Test-Time Modality Generalization (TTMG) framework, which comprises two core components: Modality-Aware Style Projection (MASP) and Modality-Sensitive Instance Whitening (MSIW), designed to enhance generalization in arbitrary unseen modality datasets. The MASP estimates the likelihood of a test instance belonging to each seen modality and maps it onto a distribution using modality-specific style bases, guiding its projection effectively. Furthermore, as high feature covariance hinders generalization to unseen modalities, the MSIW is applied during training to selectively suppress modality-sensitive information while retaining modality-invariant features. By integrating MASP and MSIW, the TTMG framework demonstrates robust generalization capabilities for medical image segmentation in unseen modalities a challenge that current methods have largely neglected. We evaluated TTMG alongside other domain generalization techniques across eleven datasets spanning four modalities (colonoscopy, ultrasound, dermoscopy, and radiology), consistently achieving superior segmentation performance across various modality combinations.

Related papers

• Test-Time Domain Generalization via Universe Learning: A Multi-Graph Matching Approach for Medical Image Segmentation [17.49123106322442]
  Test-time adaptation (TTA) adjusts a learned model using unlabeled test data. We incorporate morphological information and propose a framework based on multi-graph matching. Our method outperforms other state-of-the-art approaches on two medical image segmentation benchmarks.
  arXiv  Detail & Related papers  (2025-03-17T10:11:11Z)
• Enhancing SAM with Efficient Prompting and Preference Optimization for Semi-supervised Medical Image Segmentation [30.524999223901645]
  We propose an enhanced Segment Anything Model (SAM) framework that utilizes annotation-efficient prompts generated in a fully unsupervised fashion. We adopt the direct preference optimization technique to design an optimal policy that enables the model to generate high-fidelity segmentations. State-of-the-art performance of our framework in tasks such as lung segmentation, breast tumor segmentation, and organ segmentation across various modalities, including X-ray, ultrasound, and abdominal CT, justifies its effectiveness in low-annotation data scenarios.
  arXiv  Detail & Related papers  (2025-03-06T17:28:48Z)
• AMM-Diff: Adaptive Multi-Modality Diffusion Network for Missing Modality Imputation [2.8498944632323755]
  In clinical practice, full imaging is not always feasible, often due to complex acquisition protocols, stringent privacy regulations, or specific clinical needs.<n>A promising solution is missing data imputation, where absent modalities are generated from available ones.<n>We propose an Adaptive Multi-Modality Diffusion Network (AMM-Diff), a novel diffusion-based generative model capable of handling any number of input modalities and generating the missing ones.
  arXiv  Detail & Related papers  (2025-01-22T12:29:33Z)
• Towards Modality Generalization: A Benchmark and Prospective Analysis [56.84045461854789]
  This paper introduces Modality Generalization (MG), which focuses on enabling models to generalize to unseen modalities.<n>We propose a comprehensive benchmark featuring multi-modal algorithms and adapt existing methods that focus on generalization.<n>Our work provides a foundation for advancing robust and adaptable multi-modal models, enabling them to handle unseen modalities in realistic scenarios.
  arXiv  Detail & Related papers  (2024-12-24T08:38:35Z)
• Modality Prompts for Arbitrary Modality Salient Object Detection [57.610000247519196]
  This paper delves into the task of arbitrary modality salient object detection (AM SOD) It aims to detect salient objects from arbitrary modalities, eg RGB images, RGB-D images, and RGB-D-T images. A novel modality-adaptive Transformer (MAT) will be proposed to investigate two fundamental challenges of AM SOD.
  arXiv  Detail & Related papers  (2024-05-06T11:02:02Z)
• Exploiting Modality-Specific Features For Multi-Modal Manipulation Detection And Grounding [54.49214267905562]
  We construct a transformer-based framework for multi-modal manipulation detection and grounding tasks. Our framework simultaneously explores modality-specific features while preserving the capability for multi-modal alignment. We propose an implicit manipulation query (IMQ) that adaptively aggregates global contextual cues within each modality.
  arXiv  Detail & Related papers  (2023-09-22T06:55:41Z)
• Enhancing Modality-Agnostic Representations via Meta-Learning for Brain Tumor Segmentation [16.747365311040863]
  We propose a novel approach to learn enhanced modality-agnostic representations by employing a meta-learning strategy in training. Our framework significantly outperforms state-of-the-art brain tumor segmentation techniques in missing modality scenarios.
  arXiv  Detail & Related papers  (2023-02-08T19:53:07Z)
• Exploiting Partial Common Information Microstructure for Multi-Modal Brain Tumor Segmentation [11.583406152227637]
  Learning with multiple modalities is crucial for automated brain tumor segmentation from magnetic resonance imaging data. Existing approaches are oblivious to partial common information shared by subsets of the modalities. In this paper, we show that identifying such partial common information can significantly boost the discriminative power of image segmentation models.
  arXiv  Detail & Related papers  (2023-02-06T01:28:52Z)
• M-GenSeg: Domain Adaptation For Target Modality Tumor Segmentation With Annotation-Efficient Supervision [4.023899199756184]
  M-GenSeg is a new semi-supervised generative training strategy for cross-modality tumor segmentation. We evaluate the performance on a brain tumor segmentation dataset composed of four different contrast sequences. Unlike the prior art, M-GenSeg also introduces the ability to train with a partially annotated source modality.
  arXiv  Detail & Related papers  (2022-12-14T15:19:06Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)
• A Novel Unified Conditional Score-based Generative Framework for Multi-modal Medical Image Completion [54.512440195060584]
  We propose the Unified Multi-Modal Conditional Score-based Generative Model (UMM-CSGM) to take advantage of Score-based Generative Model (SGM) UMM-CSGM employs a novel multi-in multi-out Conditional Score Network (mm-CSN) to learn a comprehensive set of cross-modal conditional distributions. Experiments on BraTS19 dataset show that the UMM-CSGM can more reliably synthesize the heterogeneous enhancement and irregular area in tumor-induced lesions.
  arXiv  Detail & Related papers  (2022-07-07T16:57:21Z)
• Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency [24.78258331561847]
  This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired medical images. We leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios.
  arXiv  Detail & Related papers  (2022-06-21T17:50:29Z)
• Robust Multimodal Brain Tumor Segmentation via Feature Disentanglement and Gated Fusion [71.87627318863612]
  We propose a novel multimodal segmentation framework which is robust to the absence of imaging modalities. Our network uses feature disentanglement to decompose the input modalities into the modality-specific appearance code. We validate our method on the important yet challenging multimodal brain tumor segmentation task with the BRATS challenge dataset.
  arXiv  Detail & Related papers  (2020-02-22T14:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.