One-Prompt to Segment All Medical Images

• URL: http://arxiv.org/abs/2305.10300v5
• Date: Wed, 17 Apr 2024 11:04:57 GMT
• Title: One-Prompt to Segment All Medical Images
• Authors: Junde Wu, Jiayuan Zhu, Yueming Jin, Min Xu,
• Abstract summary: This paper introduces a new paradigm toward the universal medical image segmentation, termed 'One-Prompt' One-Prompt combines the strengths of one-shot and interactive methods. In the inference stage, with just textbfone prompted sample, it can adeptly handle the unseen task in a single forward pass. Tested on 14 previously unseen datasets, the One-Prompt Model showcases superior zero-shot segmentation capabilities, outperforming a wide range of related methods.
• Score: 18.829371793411347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large foundation models, known for their strong zero-shot generalization, have excelled in visual and language applications. However, applying them to medical image segmentation, a domain with diverse imaging types and target labels, remains an open challenge. Current approaches, such as adapting interactive segmentation models like Segment Anything Model (SAM), require user prompts for each sample during inference. Alternatively, transfer learning methods like few/one-shot models demand labeled samples, leading to high costs. This paper introduces a new paradigm toward the universal medical image segmentation, termed 'One-Prompt Segmentation.' One-Prompt Segmentation combines the strengths of one-shot and interactive methods. In the inference stage, with just \textbf{one prompted sample}, it can adeptly handle the unseen task in a single forward pass. We train One-Prompt Model on 64 open-source medical datasets, accompanied by the collection of over 3,000 clinician-labeled prompts. Tested on 14 previously unseen datasets, the One-Prompt Model showcases superior zero-shot segmentation capabilities, outperforming a wide range of related methods. The code and data is released as https://github.com/KidsWithTokens/one-prompt.

Related papers

• Automating MedSAM by Learning Prompts with Weak Few-Shot Supervision [10.609371657347806]
  This work proposes to replace conditioning on input prompts with a lightweight module that directly learns a prompt embedding from the image embedding. Our approach is validated on MedSAM, a version of SAM fine-tuned for medical images.
  arXiv  Detail & Related papers  (2024-09-30T13:53:01Z)
• Curriculum Prompting Foundation Models for Medical Image Segmentation [17.33821260899367]
  Adapting large pre-trained foundation models, e.g., SAM, for medical image segmentation remains a significant challenge. Past works have been heavily reliant on a singular type of prompt for each instance, necessitating manual input of an ideally correct prompt. We propose to utilize prompts of different granularity, which are sourced from original images to provide a broader scope of clinical insights. In response, we have designed a coarse-to-fine mechanism, referred to as curriculum prompting, that progressively integrates prompts of different types.
  arXiv  Detail & Related papers  (2024-09-01T11:00:18Z)
• ProtoSAM: One-Shot Medical Image Segmentation With Foundational Models [29.781228739479893]
  ProtoSAM is a new framework for one-shot medical image segmentation. It combines the use of prototypical networks, known for few-shot segmentation, with SAM - a natural image foundation model.
  arXiv  Detail & Related papers  (2024-07-09T17:04:08Z)
• Rethinking Interactive Image Segmentation with Low Latency, High Quality, and Diverse Prompts [68.86537322287474]
  Low-latency and high-quality interactive segmentation with diverse prompts are challenging for specialist and generalist models. We propose SegNext, a next-generation interactive segmentation approach offering low latency, high quality, and diverse prompt support. Our method outperforms current state-of-the-art methods on HQSeg-44K and DAVIS, both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2024-03-31T17:02:24Z)
• Unsupervised Universal Image Segmentation [59.0383635597103]
  We propose an Unsupervised Universal model (U2Seg) adept at performing various image segmentation tasks. U2Seg generates pseudo semantic labels for these segmentation tasks via leveraging self-supervised models. We then self-train the model on these pseudo semantic labels, yielding substantial performance gains.
  arXiv  Detail & Related papers  (2023-12-28T18:59:04Z)
• Self-Prompting Large Vision Models for Few-Shot Medical Image Segmentation [14.135249795318591]
  We propose a novel perspective on self-prompting in medical vision applications. We harness the embedding space of the Segment Anything Model to prompt itself through a simple yet effective linear pixel-wise classifier. We achieve competitive results on multiple datasets.
  arXiv  Detail & Related papers  (2023-08-15T08:20:07Z)
• Diffusion Models for Open-Vocabulary Segmentation [79.02153797465324]
  OVDiff is a novel method that leverages generative text-to-image diffusion models for unsupervised open-vocabulary segmentation. It relies solely on pre-trained components and outputs the synthesised segmenter directly, without training.
  arXiv  Detail & Related papers  (2023-06-15T17:51:28Z)
• Exploring Open-Vocabulary Semantic Segmentation without Human Labels [76.15862573035565]
  We present ZeroSeg, a novel method that leverages the existing pretrained vision-language model (VL) to train semantic segmentation models. ZeroSeg overcomes this by distilling the visual concepts learned by VL models into a set of segment tokens, each summarizing a localized region of the target image. Our approach achieves state-of-the-art performance when compared to other zero-shot segmentation methods under the same training data.
  arXiv  Detail & Related papers  (2023-06-01T08:47:06Z)
• Segment Everything Everywhere All at Once [124.90835636901096]
  We present SEEM, a promptable and interactive model for segmenting everything everywhere all at once in an image. We propose a novel decoding mechanism that enables diverse prompting for all types of segmentation tasks. We conduct a comprehensive empirical study to validate the effectiveness of SEEM across diverse segmentation tasks.
  arXiv  Detail & Related papers  (2023-04-13T17:59:40Z)
• Segment Anything Model (SAM) for Digital Pathology: Assess Zero-shot Segmentation on Whole Slide Imaging [12.533476185972527]
  The segment anything model (SAM) was released as a foundation model for image segmentation. We evaluate the zero-shot segmentation performance of SAM model on representative segmentation tasks on whole slide imaging (WSI) The results suggest that the zero-shot SAM model achieves remarkable segmentation performance for large connected objects.
  arXiv  Detail & Related papers  (2023-04-09T04:06:59Z)
• Prompt-Based Multi-Modal Image Segmentation [81.58378196535003]
  We propose a system that can generate image segmentations based on arbitrary prompts at test time. A prompt can be either a text or an image. We build upon the CLIP model as a backbone which we extend with a transformer-based decoder.
  arXiv  Detail & Related papers  (2021-12-18T21:27:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.