Interactive 3D Medical Image Segmentation with SAM 2

• URL: http://arxiv.org/abs/2408.02635v1
• Date: Mon, 5 Aug 2024 16:58:56 GMT
• Title: Interactive 3D Medical Image Segmentation with SAM 2
• Authors: Chuyun Shen, Wenhao Li, Yuhang Shi, Xiangfeng Wang,
• Abstract summary: We explore the zero-shot capabilities of SAM 2, the next-generation Meta SAM model trained on videos, for 3D medical image segmentation. By treating sequential 2D slices of 3D images as video frames, SAM 2 can fully automatically propagate annotations from a single frame to the entire 3D volume.
• Score: 17.523874868612577
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive medical image segmentation (IMIS) has shown significant potential in enhancing segmentation accuracy by integrating iterative feedback from medical professionals. However, the limited availability of enough 3D medical data restricts the generalization and robustness of most IMIS methods. The Segment Anything Model (SAM), though effective for 2D images, requires expensive semi-auto slice-by-slice annotations for 3D medical images. In this paper, we explore the zero-shot capabilities of SAM 2, the next-generation Meta SAM model trained on videos, for 3D medical image segmentation. By treating sequential 2D slices of 3D images as video frames, SAM 2 can fully automatically propagate annotations from a single frame to the entire 3D volume. We propose a practical pipeline for using SAM 2 in 3D medical image segmentation and present key findings highlighting its efficiency and potential for further optimization. Concretely, numerical experiments on the BraTS2020 and the medical segmentation decathlon datasets demonstrate that SAM 2 still has a gap with supervised methods but can narrow the gap in specific settings and organ types, significantly reducing the annotation burden on medical professionals. Our code will be open-sourced and available at https://github.com/Chuyun-Shen/SAM_2_Medical_3D.

Related papers

• DB-SAM: Delving into High Quality Universal Medical Image Segmentation [100.63434169944853]
  We propose a dual-branch adapted SAM framework, named DB-SAM, to bridge the gap between natural and 2D/3D medical data. Our proposed DB-SAM achieves an absolute gain of 8.8%, compared to a recent medical SAM adapter in the literature.
  arXiv  Detail & Related papers  (2024-10-05T14:36:43Z)
• SAM2Point: Segment Any 3D as Videos in Zero-shot and Promptable Manners [87.76470518069338]
  We introduce SAM2Point, a preliminary exploration adapting Segment Anything Model 2 (SAM 2) for promptable 3D segmentation. Our framework supports various prompt types, including 3D points, boxes, and masks, and can generalize across diverse scenarios, such as 3D objects, indoor scenes, sparse outdoor environments, and raw LiDAR. To our best knowledge, we present the most faithful implementation of SAM in 3D, which may serve as a starting point for future research in promptable 3D segmentation.
  arXiv  Detail & Related papers  (2024-08-29T17:59:45Z)
• A Short Review and Evaluation of SAM2's Performance in 3D CT Image Segmentation [18.543271487108868]
  We review existing benchmarks and point out that the SAM2 paper clearly outlines a zero-shot evaluation pipeline. Our findings reveal that directly applying SAM2 on 3D medical imaging in a zero-shot manner is far from satisfactory. For smaller single-connected objects like kidney and aorta, SAM2 performs reasonably well but for most organs it is still far behind state-of-the-art 3D annotation methods.
  arXiv  Detail & Related papers  (2024-08-20T22:08:42Z)
• Segment Anything in Medical Images and Videos: Benchmark and Deployment [8.51742337818826]
  We first present a comprehensive benchmarking of the Segment Anything Model 2 (SAM2) across 11 medical image modalities and videos. Then, we develop a transfer learning pipeline and demonstrate SAM2 can be quickly adapted to medical domain by fine-tuning. We implement SAM2 as a 3D slicer plugin and Gradio API for efficient 3D image and video segmentation.
  arXiv  Detail & Related papers  (2024-08-06T17:58:18Z)
• Medical SAM 2: Segment medical images as video via Segment Anything Model 2 [4.911843298581903]
  We introduce Medical SAM 2 (MedSAM-2), an advanced segmentation model that addresses both 2D and 3D medical image segmentation tasks. By adopting the philosophy of taking medical images as videos, MedSAM-2 not only applies to 3D medical images but also unlocks new One-prompt capability. Our findings show that MedSAM-2 not only surpasses existing models in performance but also exhibits superior generalization across a range of medical image segmentation tasks.
  arXiv  Detail & Related papers  (2024-08-01T18:49:45Z)
• Segment anything model 2: an application to 2D and 3D medical images [16.253160684182895]
  Segment Anything Model (SAM) has gained significant attention because of its ability to segment various objects in images given a prompt. Recently developed SAM 2 has extended this ability to video inputs. This opens an opportunity to apply SAM to 3D images, one of the fundamental tasks in the medical imaging field.
  arXiv  Detail & Related papers  (2024-08-01T17:57:25Z)
• M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Models [49.5030774873328]
  Previous research has primarily focused on 2D medical images, leaving 3D images under-explored, despite their richer spatial information. We present a large-scale 3D multi-modal medical dataset, M3D-Data, comprising 120K image-text pairs and 662K instruction-response pairs. We also introduce a new 3D multi-modal medical benchmark, M3D-Bench, which facilitates automatic evaluation across eight tasks.
  arXiv  Detail & Related papers  (2024-03-31T06:55:12Z)
• MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation [58.53672866662472]
  We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
  arXiv  Detail & Related papers  (2023-09-16T02:41:53Z)
• SAM3D: Segment Anything Model in Volumetric Medical Images [11.764867415789901]
  We introduce SAM3D, an innovative adaptation tailored for 3D volumetric medical image analysis. Unlike current SAM-based methods that segment volumetric data by converting the volume into separate 2D slices for individual analysis, our SAM3D model processes the entire 3D volume image in a unified approach.
  arXiv  Detail & Related papers  (2023-09-07T06:05:28Z)
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation [52.699139151447945]
  We propose a novel adaptation method for transferring the segment anything model (SAM) from 2D to 3D for promptable medical image segmentation. Our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation.
  arXiv  Detail & Related papers  (2023-06-23T12:09:52Z)
• Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentation [51.770805270588625]
  The Segment Anything Model (SAM) has recently gained popularity in the field of image segmentation. Recent studies and individual experiments have shown that SAM underperforms in medical image segmentation. We propose the Medical SAM Adapter (Med-SA), which incorporates domain-specific medical knowledge into the segmentation model.
  arXiv  Detail & Related papers  (2023-04-25T07:34:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.