SLiMe: Segment Like Me

• URL: http://arxiv.org/abs/2309.03179v4
• Date: Thu, 14 Mar 2024 05:15:10 GMT
• Title: SLiMe: Segment Like Me
• Authors: Aliasghar Khani, Saeid Asgari Taghanaki, Aditya Sanghi, Ali Mahdavi Amiri, Ghassan Hamarneh,
• Abstract summary: We propose SLiMe to segment images at any desired granularity using as few as one annotated sample. We carried out a knowledge-rich set of experiments examining various design factors and showed that SLiMe outperforms other existing one-shot and few-shot segmentation methods.
• Score: 24.254744102347413
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Significant strides have been made using large vision-language models, like Stable Diffusion (SD), for a variety of downstream tasks, including image editing, image correspondence, and 3D shape generation. Inspired by these advancements, we explore leveraging these extensive vision-language models for segmenting images at any desired granularity using as few as one annotated sample by proposing SLiMe. SLiMe frames this problem as an optimization task. Specifically, given a single training image and its segmentation mask, we first extract attention maps, including our novel "weighted accumulated self-attention map" from the SD prior. Then, using the extracted attention maps, the text embeddings of Stable Diffusion are optimized such that, each of them, learn about a single segmented region from the training image. These learned embeddings then highlight the segmented region in the attention maps, which in turn can then be used to derive the segmentation map. This enables SLiMe to segment any real-world image during inference with the granularity of the segmented region in the training image, using just one example. Moreover, leveraging additional training data when available, i.e. few-shot, improves the performance of SLiMe. We carried out a knowledge-rich set of experiments examining various design factors and showed that SLiMe outperforms other existing one-shot and few-shot segmentation methods.

Related papers

• Image Segmentation in Foundation Model Era: A Survey [99.19456390358211]
  Current research in image segmentation lacks a detailed analysis of distinct characteristics, challenges, and solutions associated with these advancements. This survey seeks to fill this gap by providing a thorough review of cutting-edge research centered around FM-driven image segmentation. An exhaustive overview of over 300 segmentation approaches is provided to encapsulate the breadth of current research efforts.
  arXiv  Detail & Related papers  (2024-08-23T10:07:59Z)
• Leveraging image captions for selective whole slide image annotation [0.37334049820361814]
  This paper focuses on identifying and annotating specific image regions that optimize model training. Prototype sampling is more effective than random and diversity sampling in identifying annotation regions with valuable training information. Our results show that prototype sampling is more effective than random and diversity sampling in identifying annotation regions with valuable training information.
  arXiv  Detail & Related papers  (2024-07-08T20:05:21Z)
• Leveraging Open-Vocabulary Diffusion to Camouflaged Instance Segmentation [59.78520153338878]
  Text-to-image diffusion techniques have shown exceptional capability of producing high-quality images from text descriptions. We propose a method built upon a state-of-the-art diffusion model, empowered by open-vocabulary to learn multi-scale textual-visual features for camouflaged object representations.
  arXiv  Detail & Related papers  (2023-12-29T07:59:07Z)
• Semantic-SAM: Segment and Recognize Anything at Any Granularity [83.64686655044765]
  We introduce Semantic-SAM, a universal image segmentation model to enable segment and recognize anything at any desired granularity. We consolidate multiple datasets across three granularities and introduce decoupled classification for objects and parts. For the multi-granularity capability, we propose a multi-choice learning scheme during training, enabling each click to generate masks at multiple levels.
  arXiv  Detail & Related papers  (2023-07-10T17:59:40Z)
• IFSeg: Image-free Semantic Segmentation via Vision-Language Model [67.62922228676273]
  We introduce a novel image-free segmentation task where the goal is to perform semantic segmentation given only a set of the target semantic categories. We construct this artificial training data by creating a 2D map of random semantic categories and another map of their corresponding word tokens. Our model not only establishes an effective baseline for this novel task but also demonstrates strong performances compared to existing methods.
  arXiv  Detail & Related papers  (2023-03-25T08:19:31Z)
• Progressively Dual Prior Guided Few-shot Semantic Segmentation [57.37506990980975]
  Few-shot semantic segmentation task aims at performing segmentation in query images with a few annotated support samples. We propose a progressively dual prior guided few-shot semantic segmentation network.
  arXiv  Detail & Related papers  (2022-11-20T16:19:47Z)
• Baking in the Feature: Accelerating Volumetric Segmentation by Rendering Feature Maps [42.34064154798376]
  We propose to use features extracted with models trained on large existing datasets to improve segmentation performance. We bake this feature representation into a Neural Radiance Field (NeRF) by volumetrically rendering feature maps and supervising on features extracted from each input image. Our experiments show that our method achieves higher segmentation accuracy with fewer semantic annotations than existing methods over a wide range of scenes.
  arXiv  Detail & Related papers  (2022-09-26T14:52:10Z)
• Leveraging GAN Priors for Few-Shot Part Segmentation [43.35150430895919]
  Few-shot part segmentation aims to separate different parts of an object given only a few samples. We propose to learn task-specific features in a "pre-training"-"fine-tuning" paradigm.
  arXiv  Detail & Related papers  (2022-07-27T10:17:07Z)
• Self-attention on Multi-Shifted Windows for Scene Segmentation [14.47974086177051]
  We explore the effective use of self-attention within multi-scale image windows to learn descriptive visual features. We propose three different strategies to aggregate these feature maps to decode the feature representation for dense prediction. Our models achieve very promising performance on four public scene segmentation datasets.
  arXiv  Detail & Related papers  (2022-07-10T07:36:36Z)
• SCNet: Enhancing Few-Shot Semantic Segmentation by Self-Contrastive Background Prototypes [56.387647750094466]
  Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples. Most of advanced solutions exploit a metric learning framework that performs segmentation through matching each pixel to a learned foreground prototype. This framework suffers from biased classification due to incomplete construction of sample pairs with the foreground prototype only.
  arXiv  Detail & Related papers  (2021-04-19T11:21:47Z)
• Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates [16.28285034098361]
  Weakly-supervised learning can train models by relying on weaker forms of annotation, such as scribbles. We train a multi-scale GAN to generate realistic segmentation masks at multiple resolutions, while we use scribbles to learn their correct position in the image. Central to the model's success is a novel attention gating mechanism, which we condition with adversarial signals to act as a shape prior.
  arXiv  Detail & Related papers  (2020-07-02T14:39:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.