SAM-Med3D
- URL: http://arxiv.org/abs/2310.15161v2
- Date: Sun, 29 Oct 2023 15:45:38 GMT
- Title: SAM-Med3D
- Authors: Haoyu Wang, Sizheng Guo, Jin Ye, Zhongying Deng, Junlong Cheng,
Tianbin Li, Jianpin Chen, Yanzhou Su, Ziyan Huang, Yiqing Shen, Bin Fu,
Shaoting Zhang, Junjun He, Yu Qiao
- Abstract summary: We introduce SAM-Med3D, the most comprehensive study to modify SAM for 3D medical images.
We train SAM-Med3D with over 131K 3D masks and 247 categories.
Our approach, compared with SAM, showcases pronouncedly enhanced efficiency and broad segmentation capabilities for 3D volumetric medical images.
- Score: 36.6362248184995
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Although the Segment Anything Model (SAM) has demonstrated impressive
performance in 2D natural image segmentation, its application to 3D volumetric
medical images reveals significant shortcomings, namely suboptimal performance
and unstable prediction, necessitating an excessive number of prompt points to
attain the desired outcomes. These issues can hardly be addressed by
fine-tuning SAM on medical data because the original 2D structure of SAM
neglects 3D spatial information. In this paper, we introduce SAM-Med3D, the
most comprehensive study to modify SAM for 3D medical images. Our approach is
characterized by its comprehensiveness in two primary aspects: firstly, by
comprehensively reformulating SAM to a thorough 3D architecture trained on a
comprehensively processed large-scale volumetric medical dataset; and secondly,
by providing a comprehensive evaluation of its performance. Specifically, we
train SAM-Med3D with over 131K 3D masks and 247 categories. Our SAM-Med3D
excels at capturing 3D spatial information, exhibiting competitive performance
with significantly fewer prompt points than the top-performing fine-tuned SAM
in the medical domain. We then evaluate its capabilities across 15 datasets and
analyze it from multiple perspectives, including anatomical structures,
modalities, targets, and generalization abilities. Our approach, compared with
SAM, showcases pronouncedly enhanced efficiency and broad segmentation
capabilities for 3D volumetric medical images. Our code is released at
https://github.com/uni-medical/SAM-Med3D.
Related papers
- M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models [49.5030774873328]
Previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information.
We present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs.
We also introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks.
arXiv Detail & Related papers (2024-03-31T06:55:12Z) - MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image
Segmentation [58.53672866662472]
We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM.
Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments.
By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
arXiv Detail & Related papers (2023-09-16T02:41:53Z) - SAM3D: Segment Anything Model in Volumetric Medical Images [11.764867415789901]
We introduce SAM3D, an innovative adaptation tailored for 3D volumetric medical image analysis.
Unlike current SAM-based methods that segment volumetric data by converting the volume into separate 2D slices for individual analysis, our SAM3D model processes the entire 3D volume image in a unified approach.
arXiv Detail & Related papers (2023-09-07T06:05:28Z) - SAM-Med2D [34.82072231983896]
We introduce SAM-Med2D, the most comprehensive studies on applying SAM to medical 2D images.
We first collect and curate approximately 4.6M images and 19.7M masks from public and private datasets.
We fine-tune the encoder and decoder of the original SAM to obtain a well-performed SAM-Med2D.
arXiv Detail & Related papers (2023-08-30T17:59:02Z) - 3DSAM-adapter: Holistic Adaptation of SAM from 2D to 3D for Promptable
Medical Image Segmentation [56.50064853710202]
We propose a novel adaptation method for transferring the segment anything model (SAM) from 2D to 3D for promptable medical image segmentation.
Our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation.
arXiv Detail & Related papers (2023-06-23T12:09:52Z) - SAM3D: Zero-Shot 3D Object Detection via Segment Anything Model [59.04877271899894]
This paper explores adapting the zero-shot ability of SAM to 3D object detection in this paper.
We propose a SAM-powered BEV processing pipeline to detect objects and get promising results on the large-scale open dataset.
arXiv Detail & Related papers (2023-06-04T03:09:21Z) - Segment Anything Model for Medical Images? [38.44750512574108]
The Segment Anything Model (SAM) is the first foundation model for general image segmentation.
We built a large medical segmentation dataset with 18 modalities, 84 objects, 125 object-modality paired targets, 1050K 2D images, and 6033K masks.
SAM showed remarkable performance in some specific objects but was unstable, imperfect, or even totally failed in other situations.
arXiv Detail & Related papers (2023-04-28T07:23:31Z) - Medical SAM Adapter: Adapting Segment Anything Model for Medical Image
Segmentation [51.770805270588625]
The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation.
Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation.
We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
arXiv Detail & Related papers (2023-04-25T07:34:22Z) - Segment Anything Model for Medical Image Analysis: an Experimental Study [19.95972201734614]
Segment Anything Model (SAM) is a foundation model that is intended to segment user-defined objects of interest in an interactive manner.
We evaluate SAM's ability to segment medical images on a collection of 19 medical imaging datasets from various modalities and anatomies.
arXiv Detail & Related papers (2023-04-20T17:50:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.