LarvSeg: Exploring Image Classification Data For Large Vocabulary Semantic Segmentation via Category-wise Attentive Classifier

• URL: http://arxiv.org/abs/2501.06862v1
• Date: Sun, 12 Jan 2025 16:22:17 GMT
• Title: LarvSeg: Exploring Image Classification Data For Large Vocabulary Semantic Segmentation via Category-wise Attentive Classifier
• Authors: Haojun Yu, Di Dai, Ziwei Zhao, Di He, Han Hu, Liwei Wang,
• Abstract summary: We propose a new large vocabulary semantic segmentation framework, called LarvSeg. For the first time, we provide a 21K-category semantic segmentation model with the help of ImageNet21K.
• Score: 38.75685568624425
• License:
• Abstract: Scaling up the vocabulary of semantic segmentation models is extremely challenging because annotating large-scale mask labels is labour-intensive and time-consuming. Recently, language-guided segmentation models have been proposed to address this challenge. However, their performance drops significantly when applied to out-of-distribution categories. In this paper, we propose a new large vocabulary semantic segmentation framework, called LarvSeg. Different from previous works, LarvSeg leverages image classification data to scale the vocabulary of semantic segmentation models as large-vocabulary classification datasets usually contain balanced categories and are much easier to obtain. However, for classification tasks, the category is image-level, while for segmentation we need to predict the label at pixel level. To address this issue, we first propose a general baseline framework to incorporate image-level supervision into the training process of a pixel-level segmentation model, making the trained network perform semantic segmentation on newly introduced categories in the classification data. We then observe that a model trained on segmentation data can group pixel features of categories beyond the training vocabulary. Inspired by this finding, we design a category-wise attentive classifier to apply supervision to the precise regions of corresponding categories to improve the model performance. Extensive experiments demonstrate that LarvSeg significantly improves the large vocabulary semantic segmentation performance, especially in the categories without mask labels. For the first time, we provide a 21K-category semantic segmentation model with the help of ImageNet21K. The code is available at https://github.com/HaojunYu1998/large_voc_seg.

Related papers

• USE: Universal Segment Embeddings for Open-Vocabulary Image Segmentation [33.11010205890195]
  The main challenge in open-vocabulary image segmentation now lies in accurately classifying these segments into text-defined categories. We introduce the Universal Segment Embedding (USE) framework to address this challenge. This framework is comprised of two key components: 1) a data pipeline designed to efficiently curate a large amount of segment-text pairs at various granularities, and 2) a universal segment embedding model that enables precise segment classification into a vast range of text-defined categories.
  arXiv  Detail & Related papers  (2024-06-07T21:41:18Z)
• Vocabulary-free Image Classification and Semantic Segmentation [71.78089106671581]
  We introduce the Vocabulary-free Image Classification (VIC) task, which aims to assign a class from an un-constrained language-induced semantic space to an input image without needing a known vocabulary. VIC is challenging due to the vastness of the semantic space, which contains millions of concepts, including fine-grained categories. We propose Category Search from External Databases (CaSED), a training-free method that leverages a pre-trained vision-language model and an external database.
  arXiv  Detail & Related papers  (2024-04-16T19:27:21Z)
• Synthetic Instance Segmentation from Semantic Image Segmentation Masks [15.477053085267404]
  We propose a novel paradigm called Synthetic Instance (SISeg) SISeg instance segmentation results by leveraging image masks generated by existing semantic segmentation models. In other words, the proposed model does not need extra manpower or higher computational expenses.
  arXiv  Detail & Related papers  (2023-08-02T05:13:02Z)
• Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embedding [95.78002228538841]
  We propose a new open-world semantic segmentation pipeline that makes the first attempt to learn to segment semantic objects of various open-world categories without any efforts on dense annotations. Our method can directly segment objects of arbitrary categories, outperforming zero-shot segmentation methods that require data labeling on three benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-18T09:20:04Z)
• Distilling Ensemble of Explanations for Weakly-Supervised Pre-Training of Image Segmentation Models [54.49581189337848]
  We propose a method to enable the end-to-end pre-training for image segmentation models based on classification datasets. The proposed method leverages a weighted segmentation learning procedure to pre-train the segmentation network en masse. Experiment results show that, with ImageNet accompanied by PSSL as the source dataset, the proposed end-to-end pre-training strategy successfully boosts the performance of various segmentation models.
  arXiv  Detail & Related papers  (2022-07-04T13:02:32Z)
• Semantic Segmentation In-the-Wild Without Seeing Any Segmentation Examples [34.97652735163338]
  We propose a novel approach for creating semantic segmentation masks for every object. Our method takes as input the image-level labels of the class categories present in the image. The output of this stage provides pixel-level pseudo-labels, instead of the manual pixel-level labels required by supervised methods.
  arXiv  Detail & Related papers  (2021-12-06T17:32:38Z)
• Per-Pixel Classification is Not All You Need for Semantic Segmentation [184.2905747595058]
  Mask classification is sufficiently general to solve both semantic- and instance-level segmentation tasks. We propose MaskFormer, a simple mask classification model which predicts a set of binary masks. Our method outperforms both current state-of-the-art semantic (55.6 mIoU on ADE20K) and panoptic segmentation (52.7 PQ on COCO) models.
  arXiv  Detail & Related papers  (2021-07-13T17:59:50Z)
• Segmenter: Transformer for Semantic Segmentation [79.9887988699159]
  We introduce Segmenter, a transformer model for semantic segmentation. We build on the recent Vision Transformer (ViT) and extend it to semantic segmentation. It outperforms the state of the art on the challenging ADE20K dataset and performs on-par on Pascal Context and Cityscapes.
  arXiv  Detail & Related papers  (2021-05-12T13:01:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.