Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation

• URL: http://arxiv.org/abs/2308.07624v1
• Date: Tue, 15 Aug 2023 08:20:07 GMT
• Title: Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation
• Authors: Qi Wu, Yuyao Zhang, Marawan Elbatel
• Abstract summary: We propose a novel perspective on self-prompting in medical vision applications. We harness the embedding space of the Segment Anything Model to prompt itself through a simple yet effective linear pixel-wise classifier. We achieve competitive results on multiple datasets.
• Score: 14.135249795318591
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advancements in large foundation models have shown promising potential in the medical industry due to their flexible prompting capability. One such model, the Segment Anything Model (SAM), a prompt-driven segmentation model, has shown remarkable performance improvements, surpassing state-of-the-art approaches in medical image segmentation. However, existing methods primarily rely on tuning strategies that require extensive data or prior prompts tailored to the specific task, making it particularly challenging when only a limited number of data samples are available. In this paper, we propose a novel perspective on self-prompting in medical vision applications. Specifically, we harness the embedding space of SAM to prompt itself through a simple yet effective linear pixel-wise classifier. By preserving the encoding capabilities of the large model, the contextual information from its decoder, and leveraging its interactive promptability, we achieve competitive results on multiple datasets (i.e. improvement of more than 15% compared to fine-tuning the mask decoder using a few images).

Related papers

• Prompting Segment Anything Model with Domain-Adaptive Prototype for Generalizable Medical Image Segmentation [49.5901368256326]
  We propose a novel Domain-Adaptive Prompt framework for fine-tuning the Segment Anything Model (termed as DAPSAM) in segmenting medical images. Our DAPSAM achieves state-of-the-art performance on two medical image segmentation tasks with different modalities.
  arXiv  Detail & Related papers  (2024-09-19T07:28:33Z)
• Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts [68.86537322287474]
  Low-latency and high-quality interactive segmentation with diverse prompts are challenging for specialist and generalist models. We propose SegNext, a next-generation interactive segmentation approach offering low latency, high quality, and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS, both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2024-03-31T17:02:24Z)
• Data-efficient Large Vision Models through Sequential Autoregression [58.26179273091461]
  We develop an efficient, autoregression-based vision model on a limited dataset. We demonstrate how this model achieves proficiency in a spectrum of visual tasks spanning both high-level and low-level semantic understanding. Our empirical evaluations underscore the model's agility in adapting to various tasks, heralding a significant reduction in the parameter footprint.
  arXiv  Detail & Related papers  (2024-02-07T13:41:53Z)
• VISION-MAE: A Foundation Model for Medical Image Segmentation and Classification [36.8105960525233]
  We present a novel foundation model, VISION-MAE, specifically designed for medical imaging. VISION-MAE is trained on a dataset of 2.5 million unlabeled images from various modalities. It is then adapted to classification and segmentation tasks using explicit labels.
  arXiv  Detail & Related papers  (2024-02-01T21:45:12Z)
• Segment Anything Model for Medical Image Segmentation: Current Applications and Future Directions [8.216028136706948]
  The recent introduction of the Segment Anything Model (SAM) signifies a noteworthy expansion of the prompt-driven paradigm into the domain of image segmentation. We provide a comprehensive overview of recent endeavors aimed at extending the efficacy of SAM to medical image segmentation tasks. We explore potential avenues for future research directions in SAM's role within medical image segmentation.
  arXiv  Detail & Related papers  (2024-01-07T14:25:42Z)
• TransMed: Large Language Models Enhance Vision Transformer for Biomedical Image Classification [11.202967500669402]
  Few-shot learning has been studied to adapt models to tasks with very few samples. We propose a novel approach that contextualizes labels via large language models (LLMs) Our findings reveal that the context generated by LLMs significantly enhances the discrimination of semantic embeddings for similar categories.
  arXiv  Detail & Related papers  (2023-12-12T09:58:07Z)
• Task-driven Prompt Evolution for Foundation Models [0.8192907805418581]
  We propose a plug-and-play Prompt Optimization Technique for foundation models like SAM (SAMPOT) We demonstrate the utility of SAMPOT on lung segmentation in chest X-ray images.
  arXiv  Detail & Related papers  (2023-10-26T04:08:07Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
  We propose a learnable weight-based hybrid medical image segmentation approach. Our approach is easy to integrate into any hybrid model and requires no external training data. Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-06-15T17:55:05Z)
• SAM on Medical Images: A Comprehensive Study on Three Prompt Modes [12.42280534113305]
  The Segment Anything Model (SAM) made an eye-catching debut recently and inspired many researchers to explore its potential and limitation in terms of zero-shot generalization capability. In this paper, we evaluate whether SAM has the potential to become the foundation model for medical image segmentation tasks. We also explore what kind of prompt can lead to the best zero-shot performance with different modalities.
  arXiv  Detail & Related papers  (2023-04-28T18:18:07Z)
• Ambiguous Medical Image Segmentation using Diffusion Models [60.378180265885945]
  We introduce a single diffusion model-based approach that produces multiple plausible outputs by learning a distribution over group insights. Our proposed model generates a distribution of segmentation masks by leveraging the inherent sampling process of diffusion. Comprehensive results show that our proposed approach outperforms existing state-of-the-art ambiguous segmentation networks.
  arXiv  Detail & Related papers  (2023-04-10T17:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.