Not Quite Anything: Overcoming SAMs Limitations for 3D Medical Imaging

• URL: http://arxiv.org/abs/2511.19471v1
• Date: Sat, 22 Nov 2025 05:29:27 GMT
• Title: Not Quite Anything: Overcoming SAMs Limitations for 3D Medical Imaging
• Abstract summary: Foundation segmentation models such as SAM and SAM-2 perform well on natural images but struggle with brain MRIs.<n>We propose a compositional alternative where the foundation model output is treated as an additional input channel.<n>We generate SAM-2 prompts by using a lightweight 3D U-Net that was previously trained on MRI segmentation.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation segmentation models such as SAM and SAM-2 perform well on natural images but struggle with brain MRIs where structures like the caudate and thalamus lack sharp boundaries and have low contrast. Rather than fine tune these models (for example MedSAM), we propose a compositional alternative where the foundation model output is treated as an additional input channel and passed alongside the MRI to highlight regions of interest. We generate SAM-2 prompts by using a lightweight 3D U-Net that was previously trained on MRI segmentation. The U-Net may have been trained on a different dataset, so its guesses are often imprecise but usually in the correct region. The edges of the resulting foundation model guesses are smoothed to improve alignment with the MRI. We also test prompt free segmentation using DINO attention maps in the same framework. This has-a architecture avoids modifying foundation weights and adapts to domain shift without retraining the foundation model. It reaches about 96 percent volume accuracy on basal ganglia segmentation, which is sufficient for our study of longitudinal volume change. The approach is fast, label efficient, and robust to out of distribution scans. We apply it to study inflammation linked changes in sudden onset pediatric OCD.

Related papers

• Putting the Segment Anything Model to the Test with 3D Knee MRI - A Comparison with State-of-the-Art Performance [2.167419736510938]
  Menisci are cartilaginous tissue found within the knee that contribute to joint lubrication and weight dispersal.<n> Accurate automated segmentation of menisci would allow for earlier detection and treatment of meniscal abnormalities.
  arXiv  Detail & Related papers  (2025-04-17T21:18:58Z)
• Novel adaptation of video segmentation to 3D MRI: efficient zero-shot knee segmentation with SAM2 [1.6237741047782823]
  We introduce a method for zero-shot, single-prompt segmentation of 3D knee MRI by adapting Segment Anything Model 2. By treating slices from 3D medical volumes as individual video frames, we leverage SAM2's advanced capabilities to generate motion- and spatially-aware predictions. We demonstrate that SAM2 can efficiently perform segmentation tasks in a zero-shot manner with no additional training or fine-tuning.
  arXiv  Detail & Related papers  (2024-08-08T21:39:15Z)
• Testing the Segment Anything Model on radiology data [0.0]
  The Segment Anything Model (SAM) was recently proposed and stands as the first foundation model for image segmentation. We show that while acceptable in a very limited set of cases, the overall trend implies that these models are insufficient for MRI segmentation. We note that while foundation models trained on natural images are set to become key aspects of predictive modelling, they may prove ineffective when used on other imaging modalities.
  arXiv  Detail & Related papers  (2023-12-20T09:45:21Z)
• MA-SAM: Modality-agnostic SAM Adaptation for 3D Medical Image Segmentation [58.53672866662472]
  We introduce a modality-agnostic SAM adaptation framework, named as MA-SAM. Our method roots in the parameter-efficient fine-tuning strategy to update only a small portion of weight increments. By injecting a series of 3D adapters into the transformer blocks of the image encoder, our method enables the pre-trained 2D backbone to extract third-dimensional information from input data.
  arXiv  Detail & Related papers  (2023-09-16T02:41:53Z)
• AdaptiveSAM: Towards Efficient Tuning of SAM for Surgical Scene Segmentation [49.59991322513561]
  We propose an adaptive modification of Segment-Anything (SAM) that can adjust to new datasets quickly and efficiently. AdaptiveSAM uses free-form text as prompt and can segment the object of interest with just the label name as prompt. Our experiments show that AdaptiveSAM outperforms current state-of-the-art methods on various medical imaging datasets.
  arXiv  Detail & Related papers  (2023-08-07T17:12:54Z)
• Diffusion Models for Counterfactual Generation and Anomaly Detection in Brain Images [39.94162291765236]
  We present a weakly supervised method to generate a healthy version of a diseased image and then use it to obtain a pixel-wise anomaly map. We employ a diffusion model trained on healthy samples and combine Denoising Diffusion Probabilistic Model (DDPM) and Denoising Implicit Model (DDIM) at each step of the sampling process.
  arXiv  Detail & Related papers  (2023-08-03T21:56:50Z)
• 3DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentation [52.699139151447945]
  We propose a novel adaptation method for transferring the segment anything model (SAM) from 2D to 3D for promptable medical image segmentation. Our model can outperform domain state-of-the-art medical image segmentation models on 3 out of 4 tasks, specifically by 8.25%, 29.87%, and 10.11% for kidney tumor, pancreas tumor, colon cancer segmentation, and achieve similar performance for liver tumor segmentation.
  arXiv  Detail & Related papers  (2023-06-23T12:09:52Z)
• Multiple Sclerosis Lesions Segmentation using Attention-Based CNNs in FLAIR Images [0.2578242050187029]
  Multiple Sclerosis (MS) is an autoimmune, and demyelinating disease that leads to lesions in the central nervous system. Up to now a multitude of multimodality automatic biomedical approaches is used to segment lesions. Authors propose a method employing just one modality (FLAIR image) to segment MS lesions accurately.
  arXiv  Detail & Related papers  (2022-01-05T21:37:43Z)
• Explainable multiple abnormality classification of chest CT volumes with AxialNet and HiResCAM [89.2175350956813]
  We introduce the challenging new task of explainable multiple abnormality classification in volumetric medical images. We propose a multiple instance learning convolutional neural network, AxialNet, that allows identification of top slices for each abnormality. We then aim to improve the model's learning through a novel mask loss that leverages HiResCAM and 3D allowed regions.
  arXiv  Detail & Related papers  (2021-11-24T01:14:33Z)
• Automatic size and pose homogenization with spatial transformer network to improve and accelerate pediatric segmentation [51.916106055115755]
  We propose a new CNN architecture that is pose and scale invariant thanks to the use of Spatial Transformer Network (STN) Our architecture is composed of three sequential modules that are estimated together during training. We test the proposed method in kidney and renal tumor segmentation on abdominal pediatric CT scanners.
  arXiv  Detail & Related papers  (2021-07-06T14:50:03Z)
• Rotation-Equivariant Deep Learning for Diffusion MRI [49.321304988619865]
  Convolutional networks are successful, but they have recently been outperformed by new neural networks that are equivariant under rotations and translations. Here we generalize them to 6D diffusion MRI data, ensuring joint equivariance under 3D roto-translations in image space and the matching 3D rotations in $q$-space. Our proposed neural networks yield better results and require fewer scans for training compared to non-rotation-equivariant deep learning.
  arXiv  Detail & Related papers  (2021-02-13T15:18:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.