Self-supervised learning via inter-modal reconstruction and feature projection networks for label-efficient 3D-to-2D segmentation

• URL: http://arxiv.org/abs/2307.03008v3
• Date: Thu, 13 Jul 2023 09:10:28 GMT
• Title: Self-supervised learning via inter-modal reconstruction and feature projection networks for label-efficient 3D-to-2D segmentation
• Authors: Jos\'e Morano, Guilherme Aresta, Dmitrii Lachinov, Julia Mai, Ursula Schmidt-Erfurth, Hrvoje Bogunovi\'c
• Abstract summary: We propose a novel convolutional neural network (CNN) and self-supervised learning (SSL) method for label-efficient 3D-to-2D segmentation. Results on different datasets demonstrate that the proposed CNN significantly improves the state of the art in scenarios with limited labeled data by up to 8% in Dice score.
• Score: 4.5206601127476445
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep learning has become a valuable tool for the automation of certain medical image segmentation tasks, significantly relieving the workload of medical specialists. Some of these tasks require segmentation to be performed on a subset of the input dimensions, the most common case being 3D-to-2D. However, the performance of existing methods is strongly conditioned by the amount of labeled data available, as there is currently no data efficient method, e.g. transfer learning, that has been validated on these tasks. In this work, we propose a novel convolutional neural network (CNN) and self-supervised learning (SSL) method for label-efficient 3D-to-2D segmentation. The CNN is composed of a 3D encoder and a 2D decoder connected by novel 3D-to-2D blocks. The SSL method consists of reconstructing image pairs of modalities with different dimensionality. The approach has been validated in two tasks with clinical relevance: the en-face segmentation of geographic atrophy and reticular pseudodrusen in optical coherence tomography. Results on different datasets demonstrate that the proposed CNN significantly improves the state of the art in scenarios with limited labeled data by up to 8% in Dice score. Moreover, the proposed SSL method allows further improvement of this performance by up to 23%, and we show that the SSL is beneficial regardless of the network architecture.

Related papers

• Cross-Dimensional Medical Self-Supervised Representation Learning Based on a Pseudo-3D Transformation [68.60747298865394]
  We propose a new cross-dimensional SSL framework based on a pseudo-3D transformation (CDSSL-P3D) Specifically, we introduce an image transformation based on the im2col algorithm, which converts 2D images into a format consistent with 3D data. This transformation enables seamless integration of 2D and 3D data, and facilitates cross-dimensional self-supervised learning for 3D medical image analysis.
  arXiv  Detail & Related papers  (2024-06-03T02:57:25Z)
• Simultaneous Alignment and Surface Regression Using Hybrid 2D-3D Networks for 3D Coherent Layer Segmentation of Retinal OCT Images with Full and Sparse Annotations [32.69359482975795]
  This work presents a novel framework based on hybrid 2D-3D convolutional neural networks (CNNs) to obtain continuous 3D retinal layer surfaces from OCT volumes. Experiments on a synthetic dataset and three public clinical datasets show that our framework can effectively align the B-scans for potential motion correction.
  arXiv  Detail & Related papers  (2023-12-04T08:32:31Z)
• Spatiotemporal Modeling Encounters 3D Medical Image Analysis: Slice-Shift UNet with Multi-View Fusion [0.0]
  We propose a new 2D-based model dubbed Slice SHift UNet which encodes three-dimensional features at 2D CNN's complexity. More precisely multi-view features are collaboratively learned by performing 2D convolutions along the three planes of a volume. The effectiveness of our approach is validated in Multi-Modality Abdominal Multi-Organ axis (AMOS) and Multi-Atlas Labeling Beyond the Cranial Vault (BTCV) datasets.
  arXiv  Detail & Related papers  (2023-07-24T14:53:23Z)
• Towards Label-free Scene Understanding by Vision Foundation Models [87.13117617056004]
  We investigate the potential of vision foundation models in enabling networks to comprehend 2D and 3D worlds without labelled data. We propose a novel Cross-modality Noisy Supervision (CNS) method that leverages the strengths of CLIP and SAM to supervise 2D and 3D networks simultaneously. Our 2D and 3D network achieves label-free semantic segmentation with 28.4% and 33.5% mIoU on ScanNet, improving 4.7% and 7.9%, respectively.
  arXiv  Detail & Related papers  (2023-06-06T17:57:49Z)
• Joint Self-Supervised Image-Volume Representation Learning with Intra-Inter Contrastive Clustering [31.52291149830299]
  Self-supervised learning can overcome the lack of labeled training samples by learning feature representations from unlabeled data. Most current SSL techniques in the medical field have been designed for either 2D images or 3D volumes. We propose a novel framework for unsupervised joint learning on 2D and 3D data modalities.
  arXiv  Detail & Related papers  (2022-12-04T18:57:44Z)
• Two-Stream Graph Convolutional Network for Intra-oral Scanner Image Segmentation [133.02190910009384]
  We propose a two-stream graph convolutional network (i.e., TSGCN) to handle inter-view confusion between different raw attributes. Our TSGCN significantly outperforms state-of-the-art methods in 3D tooth (surface) segmentation.
  arXiv  Detail & Related papers  (2022-04-19T10:41:09Z)
• Multi-Slice Dense-Sparse Learning for Efficient Liver and Tumor Segmentation [4.150096314396549]
  Deep convolutional neural network (DCNNs) has obtained tremendous success in 2D and 3D medical image segmentation. We propose a novel dense-sparse training flow from a data perspective, in which, densely adjacent slices and sparsely adjacent slices are extracted as inputs for regularizing DCNNs. We also design a 2.5D light-weight nnU-Net from a network perspective, in which, depthwise separable convolutions are adopted to improve the efficiency.
  arXiv  Detail & Related papers  (2021-08-15T15:29:48Z)
• TSGCNet: Discriminative Geometric Feature Learning with Two-Stream GraphConvolutional Network for 3D Dental Model Segmentation [141.2690520327948]
  We propose a two-stream graph convolutional network (TSGCNet) to learn multi-view information from different geometric attributes. We evaluate our proposed TSGCNet on a real-patient dataset of dental models acquired by 3D intraoral scanners.
  arXiv  Detail & Related papers  (2020-12-26T08:02:56Z)
• Efficient embedding network for 3D brain tumor segmentation [0.33727511459109777]
  In this paper, we investigate a way to transfer the performance of a two-dimensional classiffication network for the purpose of three-dimensional semantic segmentation of brain tumors. As the input data is in 3D, the first layers of the encoder are devoted to the reduction of the third dimension in order to fit the input of the EfficientNet network. Experimental results on validation and test data from the BraTS 2020 challenge demonstrate that the proposed method achieve promising performance.
  arXiv  Detail & Related papers  (2020-11-22T16:17:29Z)
• 3D medical image segmentation with labeled and unlabeled data using autoencoders at the example of liver segmentation in CT images [58.720142291102135]
  This work investigates the potential of autoencoder-extracted features to improve segmentation with a convolutional neural network. A convolutional autoencoder was used to extract features from unlabeled data and a multi-scale, fully convolutional CNN was used to perform the target task of 3D liver segmentation in CT images.
  arXiv  Detail & Related papers  (2020-03-17T20:20:43Z)
• 2.75D: Boosting learning by representing 3D Medical imaging to 2D features for small data [54.223614679807994]
  3D convolutional neural networks (CNNs) have started to show superior performance to 2D CNNs in numerous deep learning tasks. Applying transfer learning on 3D CNN is challenging due to a lack of publicly available pre-trained 3D models. In this work, we proposed a novel 2D strategical representation of volumetric data, namely 2.75D. As a result, 2D CNN networks can also be used to learn volumetric information.
  arXiv  Detail & Related papers  (2020-02-11T08:24:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.