From Pixel to Patch: Synthesize Context-aware Features for Zero-shot Semantic Segmentation

• URL: http://arxiv.org/abs/2009.12232v4
• Date: Fri, 21 Jan 2022 12:38:13 GMT
• Title: From Pixel to Patch: Synthesize Context-aware Features for Zero-shot Semantic Segmentation
• Authors: Zhangxuan Gu, Siyuan Zhou, Li Niu, Zihan Zhao, Liqing Zhang
• Abstract summary: We focus on zero-shot semantic segmentation, which aims to segment unseen objects with only category-level semantic representations. We propose a novel Context-aware feature Generation Network (CaGNet), which can synthesize context-aware pixel-wise visual features for unseen categories. Experimental results on Pascal-VOC, Pascal-Context, and COCO-stuff show that our method significantly outperforms the existing zero-shot semantic segmentation methods.
• Score: 22.88452754438478
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Zero-shot learning has been actively studied for image classification task to relieve the burden of annotating image labels. Interestingly, semantic segmentation task requires more labor-intensive pixel-wise annotation, but zero-shot semantic segmentation has only attracted limited research interest. Thus, we focus on zero-shot semantic segmentation, which aims to segment unseen objects with only category-level semantic representations provided for unseen categories. In this paper, we propose a novel Context-aware feature Generation Network (CaGNet), which can synthesize context-aware pixel-wise visual features for unseen categories based on category-level semantic representations and pixel-wise contextual information. The synthesized features are used to finetune the classifier to enable segmenting unseen objects. Furthermore, we extend pixel-wise feature generation and finetuning to patch-wise feature generation and finetuning, which additionally considers inter-pixel relationship. Experimental results on Pascal-VOC, Pascal-Context, and COCO-stuff show that our method significantly outperforms the existing zero-shot semantic segmentation methods. Code is available at https://github.com/bcmi/CaGNetv2-Zero-Shot-Semantic-Segmentation.

Related papers

• Learning Semantic Segmentation with Query Points Supervision on Aerial Images [57.09251327650334]
  We present a weakly supervised learning algorithm to train semantic segmentation algorithms. Our proposed approach performs accurate semantic segmentation and improves efficiency by significantly reducing the cost and time required for manual annotation.
  arXiv  Detail & Related papers  (2023-09-11T14:32:04Z)
• Primitive Generation and Semantic-related Alignment for Universal Zero-Shot Segmentation [13.001629605405954]
  We study universal zero-shot segmentation in this work to achieve panoptic, instance, and semantic segmentation for novel categories without any training samples. We introduce a generative model to synthesize features for unseen categories, which links semantic and visual spaces. The proposed approach achieves impressively state-of-the-art performance on zero-shot panoptic segmentation, instance segmentation, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-06-19T17:59:16Z)
• Exploring Open-Vocabulary Semantic Segmentation without Human Labels [76.15862573035565]
  We present ZeroSeg, a novel method that leverages the existing pretrained vision-language model (VL) to train semantic segmentation models. ZeroSeg overcomes this by distilling the visual concepts learned by VL models into a set of segment tokens, each summarizing a localized region of the target image. Our approach achieves state-of-the-art performance when compared to other zero-shot segmentation methods under the same training data.
  arXiv  Detail & Related papers  (2023-06-01T08:47:06Z)
• Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding [95.78002228538841]
  We propose a new open-world semantic segmentation pipeline that makes the first attempt to learn to segment semantic objects of various open-world categories without any efforts on dense annotations. Our method can directly segment objects of arbitrary categories, outperforming zero-shot segmentation methods that require data labeling on three benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-18T09:20:04Z)
• Language-driven Semantic Segmentation [88.21498323896475]
  We present LSeg, a novel model for language-driven semantic image segmentation. We use a text encoder to compute embeddings of descriptive input labels. The encoder is trained with a contrastive objective to align pixel embeddings to the text embedding of the corresponding semantic class.
  arXiv  Detail & Related papers  (2022-01-10T18:59:10Z)
• Maximize the Exploration of Congeneric Semantics for Weakly Supervised Semantic Segmentation [27.155133686127474]
  We construct a graph neural network (P-GNN) based on the self-detected patches from different images that contain the same class labels. We conduct experiments on the popular PASCAL VOC 2012 benchmarks, and our model yields state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-10-08T08:59:16Z)
• Universal Weakly Supervised Segmentation by Pixel-to-Segment Contrastive Learning [28.498782661888775]
  We formulate weakly supervised segmentation as a semi-supervised metric learning problem. We propose 4 types of contrastive relationships between pixels and segments in the feature space. We deliver a universal weakly supervised segmenter with significant gains on Pascal VOC and DensePose.
  arXiv  Detail & Related papers  (2021-05-03T15:49:01Z)
• Exploring Cross-Image Pixel Contrast for Semantic Segmentation [130.22216825377618]
  We propose a pixel-wise contrastive framework for semantic segmentation in the fully supervised setting. The core idea is to enforce pixel embeddings belonging to a same semantic class to be more similar than embeddings from different classes. Our method can be effortlessly incorporated into existing segmentation frameworks without extra overhead during testing.
  arXiv  Detail & Related papers  (2021-01-28T11:35:32Z)
• Context-aware Feature Generation for Zero-shot Semantic Segmentation [18.37777970377439]
  We propose a novel context-aware feature generation method for zero-shot segmentation named CaGNet. Our method achieves state-of-the-art results on three benchmark datasets for zero-shot segmentation.
  arXiv  Detail & Related papers  (2020-08-16T12:20:49Z)
• Part-aware Prototype Network for Few-shot Semantic Segmentation [50.581647306020095]
  We propose a novel few-shot semantic segmentation framework based on the prototype representation. Our key idea is to decompose the holistic class representation into a set of part-aware prototypes. We develop a novel graph neural network model to generate and enhance the proposed part-aware prototypes.
  arXiv  Detail & Related papers  (2020-07-13T11:03:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.