A Simple Image Segmentation Framework via In-Context Examples

• URL: http://arxiv.org/abs/2410.04842v2
• Date: Tue, 8 Oct 2024 12:38:44 GMT
• Title: A Simple Image Segmentation Framework via In-Context Examples
• Authors: Yang Liu, Chenchen Jing, Hengtao Li, Muzhi Zhu, Hao Chen, Xinlong Wang, Chunhua Shen,
• Abstract summary: We present SINE, a simple image framework utilizing in-context examples. We introduce an In-context Interaction module to complement in-context information and produce correlations between the target image and the in-context example. Experiments on various segmentation tasks show the effectiveness of the proposed method.
• Score: 59.319920526160466
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recently, there have been explorations of generalist segmentation models that can effectively tackle a variety of image segmentation tasks within a unified in-context learning framework. However, these methods still struggle with task ambiguity in in-context segmentation, as not all in-context examples can accurately convey the task information. In order to address this issue, we present SINE, a simple image Segmentation framework utilizing in-context examples. Our approach leverages a Transformer encoder-decoder structure, where the encoder provides high-quality image representations, and the decoder is designed to yield multiple task-specific output masks to effectively eliminate task ambiguity. Specifically, we introduce an In-context Interaction module to complement in-context information and produce correlations between the target image and the in-context example and a Matching Transformer that uses fixed matching and a Hungarian algorithm to eliminate differences between different tasks. In addition, we have further perfected the current evaluation system for in-context image segmentation, aiming to facilitate a holistic appraisal of these models. Experiments on various segmentation tasks show the effectiveness of the proposed method.

Related papers

• Visual Prompt Selection for In-Context Learning Segmentation [77.15684360470152]
  In this paper, we focus on rethinking and improving the example selection strategy. We first demonstrate that ICL-based segmentation models are sensitive to different contexts. Furthermore, empirical evidence indicates that the diversity of contextual prompts plays a crucial role in guiding segmentation.
  arXiv  Detail & Related papers  (2024-07-14T15:02:54Z)
• SEGIC: Unleashing the Emergent Correspondence for In-Context Segmentation [87.18373801829314]
  In-context segmentation aims at segmenting novel images using a few labeled example images, termed as "in-context examples" We propose SEGIC, an end-to-end segment-in-context framework built upon a single vision foundation model (VFM) SEGIC is a straightforward yet effective approach that yields state-of-the-art performance on one-shot segmentation benchmarks.
  arXiv  Detail & Related papers  (2023-11-24T18:59:42Z)
• AIMS: All-Inclusive Multi-Level Segmentation [93.5041381700744]
  We propose a new task, All-Inclusive Multi-Level (AIMS), which segments visual regions into three levels: part, entity, and relation. We also build a unified AIMS model through multi-dataset multi-task training to address the two major challenges of annotation inconsistency and task correlation.
  arXiv  Detail & Related papers  (2023-05-28T16:28:49Z)
• CAT-Seg: Cost Aggregation for Open-Vocabulary Semantic Segmentation [56.58365347854647]
  We introduce a novel cost-based approach to adapt vision-language foundation models, notably CLIP. Our method potently adapts CLIP for segmenting seen and unseen classes by fine-tuning its encoders.
  arXiv  Detail & Related papers  (2023-03-21T12:28:21Z)
• SCNet: Enhancing Few-Shot Semantic Segmentation by Self-Contrastive Background Prototypes [56.387647750094466]
  Few-shot semantic segmentation aims to segment novel-class objects in a query image with only a few annotated examples. Most of advanced solutions exploit a metric learning framework that performs segmentation through matching each pixel to a learned foreground prototype. This framework suffers from biased classification due to incomplete construction of sample pairs with the foreground prototype only.
  arXiv  Detail & Related papers  (2021-04-19T11:21:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.