Unpaired Multi-modal Segmentation via Knowledge Distillation

• URL: http://arxiv.org/abs/2001.03111v1
• Date: Mon, 6 Jan 2020 20:03:17 GMT
• Title: Unpaired Multi-modal Segmentation via Knowledge Distillation
• Authors: Qi Dou, Quande Liu, Pheng Ann Heng, Ben Glocker
• Abstract summary: We propose a novel learning scheme for unpaired cross-modality image segmentation. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI. We have extensively validated our approach on two multi-class segmentation problems.
• Score: 77.39798870702174
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-modal learning is typically performed with network architectures containing modality-specific layers and shared layers, utilizing co-registered images of different modalities. We propose a novel learning scheme for unpaired cross-modality image segmentation, with a highly compact architecture achieving superior segmentation accuracy. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI, and only employ modality-specific internal normalization layers which compute respective statistics. To effectively train such a highly compact model, we introduce a novel loss term inspired by knowledge distillation, by explicitly constraining the KL-divergence of our derived prediction distributions between modalities. We have extensively validated our approach on two multi-class segmentation problems: i) cardiac structure segmentation, and ii) abdominal organ segmentation. Different network settings, i.e., 2D dilated network and 3D U-net, are utilized to investigate our method's general efficacy. Experimental results on both tasks demonstrate that our novel multi-modal learning scheme consistently outperforms single-modal training and previous multi-modal approaches.

Related papers

• General-Purpose Multimodal Transformer meets Remote Sensing Semantic Segmentation [35.100738362291416]
  Multimodal AI seeks to exploit complementary data sources, particularly for complex tasks like semantic segmentation. Recent trends in general-purpose multimodal networks have shown great potential to achieve state-of-the-art performance. We propose a UNet-inspired module that employs 3D convolution to encode vital local information and learn cross-modal features simultaneously.
  arXiv  Detail & Related papers  (2023-07-07T04:58:34Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• Routing with Self-Attention for Multimodal Capsule Networks [108.85007719132618]
  We present a new multimodal capsule network that allows us to leverage the strength of capsules in the context of a multimodal learning framework. To adapt the capsules to large-scale input data, we propose a novel routing by self-attention mechanism that selects relevant capsules. This allows not only for robust training with noisy video data, but also to scale up the size of the capsule network compared to traditional routing methods.
  arXiv  Detail & Related papers  (2021-12-01T19:01:26Z)
• Knowledge distillation from multi-modal to mono-modal segmentation networks [13.213798509506272]
  We propose KD-Net, a framework to transfer knowledge from a trained multi-modal network (teacher) to a mono-modal one (student) We show that the student network effectively learns from the teacher and always outperforms the baseline mono-modal network in terms of segmentation accuracy.
  arXiv  Detail & Related papers  (2021-06-17T14:46:57Z)
• Deep Class-Specific Affinity-Guided Convolutional Network for Multimodal Unpaired Image Segmentation [7.021001169318551]
  Multi-modal medical image segmentation plays an essential role in clinical diagnosis. It remains challenging as the input modalities are often not well-aligned spatially. We propose an affinity-guided fully convolutional network for multimodal image segmentation.
  arXiv  Detail & Related papers  (2021-01-05T13:56:51Z)
• Towards Cross-modality Medical Image Segmentation with Online Mutual Knowledge Distillation [71.89867233426597]
  In this paper, we aim to exploit the prior knowledge learned from one modality to improve the segmentation performance on another modality. We propose a novel Mutual Knowledge Distillation scheme to thoroughly exploit the modality-shared knowledge. Experimental results on the public multi-class cardiac segmentation data, i.e., MMWHS 2017, show that our method achieves large improvements on CT segmentation.
  arXiv  Detail & Related papers  (2020-10-04T10:25:13Z)
• MetricUNet: Synergistic Image- and Voxel-Level Learning for Precise CT Prostate Segmentation via Online Sampling [66.01558025094333]
  We propose a two-stage framework, with the first stage to quickly localize the prostate region and the second stage to precisely segment the prostate. We introduce a novel online metric learning module through voxel-wise sampling in the multi-task network. Our method can effectively learn more representative voxel-level features compared with the conventional learning methods with cross-entropy or Dice loss.
  arXiv  Detail & Related papers  (2020-05-15T10:37:02Z)
• Deep Multimodal Neural Architecture Search [178.35131768344246]
  We devise a generalized deep multimodal neural architecture search (MMnas) framework for various multimodal learning tasks. Given multimodal input, we first define a set of primitive operations, and then construct a deep encoder-decoder based unified backbone. On top of the unified backbone, we attach task-specific heads to tackle different multimodal learning tasks.
  arXiv  Detail & Related papers  (2020-04-25T07:00:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.