Label-Efficient 3D Brain Segmentation via Complementary 2D Diffusion Models with Orthogonal Views

• URL: http://arxiv.org/abs/2407.12329v1
• Date: Wed, 17 Jul 2024 06:14:53 GMT
• Title: Label-Efficient 3D Brain Segmentation via Complementary 2D Diffusion Models with Orthogonal Views
• Authors: Jihoon Cho, Suhyun Ahn, Beomju Kim, Hyungjoon Bae, Xiaofeng Liu, Fangxu Xing, Kyungeun Lee, Georges Elfakhri, Van Wedeen, Jonghye Woo, Jinah Park,
• Abstract summary: We propose a novel 3D brain segmentation approach using complementary 2D diffusion models. Our goal is to achieve reliable segmentation quality without requiring complete labels for each individual subject.
• Score: 10.944692719150071
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning-based segmentation techniques have shown remarkable performance in brain segmentation, yet their success hinges on the availability of extensive labeled training data. Acquiring such vast datasets, however, poses a significant challenge in many clinical applications. To address this issue, in this work, we propose a novel 3D brain segmentation approach using complementary 2D diffusion models. The core idea behind our approach is to first mine 2D features with semantic information extracted from the 2D diffusion models by taking orthogonal views as input, followed by fusing them into a 3D contextual feature representation. Then, we use these aggregated features to train multi-layer perceptrons to classify the segmentation labels. Our goal is to achieve reliable segmentation quality without requiring complete labels for each individual subject. Our experiments on training in brain subcortical structure segmentation with a dataset from only one subject demonstrate that our approach outperforms state-of-the-art self-supervised learning methods. Further experiments on the minimum requirement of annotation by sparse labeling yield promising results even with only nine slices and a labeled background region.

Related papers

• Augmented Efficiency: Reducing Memory Footprint and Accelerating Inference for 3D Semantic Segmentation through Hybrid Vision [9.96433151449016]
  This paper introduces a novel approach to 3D semantic segmentation, distinguished by incorporating a hybrid blend of 2D and 3D computer vision techniques. We conduct 2D semantic segmentation on RGB images linked to 3D point clouds and extend the results to 3D using an extrusion technique for specific class labels. This model serves as the current state-of-the-art 3D semantic segmentation model on the KITTI-360 dataset.
  arXiv  Detail & Related papers  (2024-07-23T00:04:10Z)
• Enhancing Single-Slice Segmentation with 3D-to-2D Unpaired Scan Distillation [21.69523493833432]
  We propose a novel 3D-to-2D distillation framework, leveraging pre-trained 3D models to enhance 2D single-slice segmentation. Unlike traditional knowledge distillation methods that require the same data input, our approach employs unpaired 3D CT scans with any contrast to guide the 2D student model.
  arXiv  Detail & Related papers  (2024-06-18T04:06:02Z)
• DatasetNeRF: Efficient 3D-aware Data Factory with Generative Radiance Fields [73.97131748433212]
  This paper introduces a novel approach capable of generating infinite, high-quality 3D-consistent 2D annotations alongside 3D point cloud segmentations. We leverage the strong semantic prior within a 3D generative model to train a semantic decoder. Once trained, the decoder efficiently generalizes across the latent space, enabling the generation of infinite data.
  arXiv  Detail & Related papers  (2023-11-18T21:58:28Z)
• Leveraging Large-Scale Pretrained Vision Foundation Models for Label-Efficient 3D Point Cloud Segmentation [67.07112533415116]
  We present a novel framework that adapts various foundational models for the 3D point cloud segmentation task. Our approach involves making initial predictions of 2D semantic masks using different large vision models. To generate robust 3D semantic pseudo labels, we introduce a semantic label fusion strategy that effectively combines all the results via voting.
  arXiv  Detail & Related papers  (2023-11-03T15:41:15Z)
• Weakly Supervised 3D Open-vocabulary Segmentation [104.07740741126119]
  We tackle the challenges in 3D open-vocabulary segmentation by exploiting pre-trained foundation models CLIP and DINO in a weakly supervised manner. We distill the open-vocabulary multimodal knowledge and object reasoning capability of CLIP and DINO into a neural radiance field (NeRF) A notable aspect of our approach is that it does not require any manual segmentation annotations for either the foundation models or the distillation process.
  arXiv  Detail & Related papers  (2023-05-23T14:16:49Z)
• You Only Need One Thing One Click: Self-Training for Weakly Supervised 3D Scene Understanding [107.06117227661204]
  We propose One Thing One Click'', meaning that the annotator only needs to label one point per object. We iteratively conduct the training and label propagation, facilitated by a graph propagation module. Our model can be compatible to 3D instance segmentation equipped with a point-clustering strategy.
  arXiv  Detail & Related papers  (2023-03-26T13:57:00Z)
• Seismic Fault Segmentation via 3D-CNN Training by a Few 2D Slices Labels [6.963867115353744]
  We present a new binary cross-entropy and smooth L1 loss to train 3D-CNN by sampling some 2D slices from 3D seismic data. Experiments show that our method can extract 3D seismic features from a few 2D slices labels on real data, to segment a complete fault volume.
  arXiv  Detail & Related papers  (2021-05-09T07:13:40Z)
• One Thing One Click: A Self-Training Approach for Weakly Supervised 3D Semantic Segmentation [78.36781565047656]
  We propose "One Thing One Click," meaning that the annotator only needs to label one point per object. We iteratively conduct the training and label propagation, facilitated by a graph propagation module. Our results are also comparable to those of the fully supervised counterparts.
  arXiv  Detail & Related papers  (2021-04-06T02:27:25Z)
• 3D Guided Weakly Supervised Semantic Segmentation [27.269847900950943]
  We propose a weakly supervised 2D semantic segmentation model by incorporating sparse bounding box labels with available 3D information. We manually labeled a subset of the 2D-3D Semantics(2D-3D-S) dataset with bounding boxes, and introduce our 2D-3D inference module to generate accurate pixel-wise segment proposal masks.
  arXiv  Detail & Related papers  (2020-12-01T03:34:15Z)
• Bidirectional RNN-based Few Shot Learning for 3D Medical Image Segmentation [11.873435088539459]
  We propose a 3D few shot segmentation framework for accurate organ segmentation using limited training samples of the target organ annotation. A U-Net like network is designed to predict segmentation by learning the relationship between 2D slices of support data and a query image. We evaluate our proposed model using three 3D CT datasets with annotations of different organs.
  arXiv  Detail & Related papers  (2020-11-19T01:44:55Z)
• 3D medical image segmentation with labeled and unlabeled data using autoencoders at the example of liver segmentation in CT images [58.720142291102135]
  This work investigates the potential of autoencoder-extracted features to improve segmentation with a convolutional neural network. A convolutional autoencoder was used to extract features from unlabeled data and a multi-scale, fully convolutional CNN was used to perform the target task of 3D liver segmentation in CT images.
  arXiv  Detail & Related papers  (2020-03-17T20:20:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.