Models Genesis
- URL: http://arxiv.org/abs/2004.07882v4
- Date: Wed, 16 Dec 2020 19:58:08 GMT
- Title: Models Genesis
- Authors: Zongwei Zhou, Vatsal Sodha, Jiaxuan Pang, Michael B. Gotway, Jianming
Liang
- Abstract summary: Transfer learning from natural images to medical images has been established as one of the most practical paradigms in deep learning for medical image analysis.
To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis.
Our experiments demonstrate that our Models Genesis significantly outperform learning from scratch and existing pre-trained 3D models in all five target 3D applications.
- Score: 10.929445262793116
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Transfer learning from natural images to medical images has been established
as one of the most practical paradigms in deep learning for medical image
analysis. To fit this paradigm, however, 3D imaging tasks in the most prominent
imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D,
losing rich 3D anatomical information, thereby inevitably compromising its
performance. To overcome this limitation, we have built a set of models, called
Generic Autodidactic Models, nicknamed Models Genesis, because they are created
ex nihilo (with no manual labeling), self-taught (learnt by self-supervision),
and generic (served as source models for generating application-specific target
models). Our extensive experiments demonstrate that our Models Genesis
significantly outperform learning from scratch and existing pre-trained 3D
models in all five target 3D applications covering both segmentation and
classification. More importantly, learning a model from scratch simply in 3D
may not necessarily yield performance better than transfer learning from
ImageNet in 2D, but our Models Genesis consistently top any 2D/2.5D approaches
including fine-tuning the models pre-trained from ImageNet as well as
fine-tuning the 2D versions of our Models Genesis, confirming the importance of
3D anatomical information and significance of Models Genesis for 3D medical
imaging. This performance is attributed to our unified self-supervised learning
framework, built on a simple yet powerful observation: the sophisticated and
recurrent anatomy in medical images can serve as strong yet free supervision
signals for deep models to learn common anatomical representation automatically
via self-supervision. As open science, all codes and pre-trained Models Genesis
are available at https://github.com/MrGiovanni/ModelsGenesis.
Related papers
- VISTA3D: Versatile Imaging SegmenTation and Annotation model for 3D Computed Tomography [18.111368889931885]
We discuss what features a 3D CT segmentation foundation model should have, and introduce VISTA3D, Versatile Imaging SegmenTation and.
The model is trained systematically on 11454 volumes encompassing 127 types of human anatomical structures and various lesions.
The model's design also achieves state-of-the-art zero-shot interactive segmentation in 3D.
arXiv Detail & Related papers (2024-06-07T22:41:39Z) - Learning the 3D Fauna of the Web [70.01196719128912]
We develop 3D-Fauna, an approach that learns a pan-category deformable 3D animal model for more than 100 animal species jointly.
One crucial bottleneck of modeling animals is the limited availability of training data.
We show that prior category-specific attempts fail to generalize to rare species with limited training images.
arXiv Detail & Related papers (2024-01-04T18:32:48Z) - Leveraging Large-Scale Pretrained Vision Foundation Models for
Label-Efficient 3D Point Cloud Segmentation [67.07112533415116]
We present a novel framework that adapts various foundational models for the 3D point cloud segmentation task.
Our approach involves making initial predictions of 2D semantic masks using different large vision models.
To generate robust 3D semantic pseudo labels, we introduce a semantic label fusion strategy that effectively combines all the results via voting.
arXiv Detail & Related papers (2023-11-03T15:41:15Z) - Promise:Prompt-driven 3D Medical Image Segmentation Using Pretrained
Image Foundation Models [13.08275555017179]
We propose ProMISe, a prompt-driven 3D medical image segmentation model using only a single point prompt.
We evaluate our model on two public datasets for colon and pancreas tumor segmentations.
arXiv Detail & Related papers (2023-10-30T16:49:03Z) - PonderV2: Pave the Way for 3D Foundation Model with A Universal
Pre-training Paradigm [114.47216525866435]
We introduce a novel universal 3D pre-training framework designed to facilitate the acquisition of efficient 3D representation.
For the first time, PonderV2 achieves state-of-the-art performance on 11 indoor and outdoor benchmarks, implying its effectiveness.
arXiv Detail & Related papers (2023-10-12T17:59:57Z) - Take-A-Photo: 3D-to-2D Generative Pre-training of Point Cloud Models [97.58685709663287]
generative pre-training can boost the performance of fundamental models in 2D vision.
In 3D vision, the over-reliance on Transformer-based backbones and the unordered nature of point clouds have restricted the further development of generative pre-training.
We propose a novel 3D-to-2D generative pre-training method that is adaptable to any point cloud model.
arXiv Detail & Related papers (2023-07-27T16:07:03Z) - AG3D: Learning to Generate 3D Avatars from 2D Image Collections [96.28021214088746]
We propose a new adversarial generative model of realistic 3D people from 2D images.
Our method captures shape and deformation of the body and loose clothing by adopting a holistic 3D generator.
We experimentally find that our method outperforms previous 3D- and articulation-aware methods in terms of geometry and appearance.
arXiv Detail & Related papers (2023-05-03T17:56:24Z) - HoloDiffusion: Training a 3D Diffusion Model using 2D Images [71.1144397510333]
We introduce a new diffusion setup that can be trained, end-to-end, with only posed 2D images for supervision.
We show that our diffusion models are scalable, train robustly, and are competitive in terms of sample quality and fidelity to existing approaches for 3D generative modeling.
arXiv Detail & Related papers (2023-03-29T07:35:56Z) - Oral-3Dv2: 3D Oral Reconstruction from Panoramic X-Ray Imaging with
Implicit Neural Representation [3.8215162658168524]
Oral-3Dv2 is a non-adversarial-learning-based model in 3D radiology reconstruction from a single panoramic X-ray image.
Our model learns to represent the 3D oral structure in an implicit way by mapping 2D coordinates into density values of voxels in the 3D space.
To the best of our knowledge, this is the first work of a non-adversarial-learning-based model in 3D radiology reconstruction from a single panoramic X-ray image.
arXiv Detail & Related papers (2023-03-21T18:17:27Z) - Transferring Models Trained on Natural Images to 3D MRI via Position
Encoded Slice Models [14.42534860640976]
2D-Slice-CNN architecture embeds all the MRI slices with 2D encoders that take 2D image input and combines them via permutation-invariant layers.
With the insight that pretrained model can serve as the 2D encoder, we initialize the 2D encoder with ImageNet pretrained weights that outperform those and trained from scratch on two neuroimaging tasks.
arXiv Detail & Related papers (2023-03-02T18:52:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.