Co-Segmentation without any Pixel-level Supervision with Application to Large-Scale Sketch Classification

• URL: http://arxiv.org/abs/2410.13582v1
• Date: Thu, 17 Oct 2024 14:16:45 GMT
• Title: Co-Segmentation without any Pixel-level Supervision with Application to Large-Scale Sketch Classification
• Authors: Nikolaos-Antonios Ypsilantis, Ondřej Chum,
• Abstract summary: We propose a novel method for object co-segmentation, i.e. pixel-level localization of a common object in a set of images. The method achieves state-of-the-art performance among methods trained with the same level of supervision. The benefits of the proposed co-segmentation method are further demonstrated in the task of large-scale sketch recognition.
• Score: 3.3104978705632777
• License:
• Abstract: This work proposes a novel method for object co-segmentation, i.e. pixel-level localization of a common object in a set of images, that uses no pixel-level supervision for training. Two pre-trained Vision Transformer (ViT) models are exploited: ImageNet classification-trained ViT, whose features are used to estimate rough object localization through intra-class token relevance, and a self-supervised DINO-ViT for intra-image token relevance. On recent challenging benchmarks, the method achieves state-of-the-art performance among methods trained with the same level of supervision (image labels) while being competitive with methods trained with pixel-level supervision (binary masks). The benefits of the proposed co-segmentation method are further demonstrated in the task of large-scale sketch recognition, that is, the classification of sketches into a wide range of categories. The limited amount of hand-drawn sketch training data is leveraged by exploiting readily available image-level-annotated datasets of natural images containing a large number of classes. To bridge the domain gap, the classifier is trained on a sketch-like proxy domain derived from edges detected on natural images. We show that sketch recognition significantly benefits when the classifier is trained on sketch-like structures extracted from the co-segmented area rather than from the full image. Code: https://github.com/nikosips/CBNC .

Related papers

• Towards Open-Vocabulary Semantic Segmentation Without Semantic Labels [53.8817160001038]
  We propose a novel method, PixelCLIP, to adapt the CLIP image encoder for pixel-level understanding. To address the challenges of leveraging masks without semantic labels, we devise an online clustering algorithm. PixelCLIP shows significant performance improvements over CLIP and competitive results compared to caption-supervised methods.
  arXiv  Detail & Related papers  (2024-09-30T01:13:03Z)
• Exploring Multi-view Pixel Contrast for General and Robust Image Forgery Localization [4.8454936010479335]
  We propose a Multi-view Pixel-wise Contrastive algorithm (MPC) for image forgery localization. Specifically, we first pre-train the backbone network with the supervised contrastive loss. Then the localization head is fine-tuned using the cross-entropy loss, resulting in a better pixel localizer.
  arXiv  Detail & Related papers  (2024-06-19T13:51:52Z)
• Pixel-Level Clustering Network for Unsupervised Image Segmentation [3.69853388955692]
  We present a pixel-level clustering framework for segmenting images into regions without using ground truth annotations. We also propose a training strategy that utilizes intra-consistency within each superpixel, inter-similarity/dissimilarity between neighboring superpixels, and structural similarity between images.
  arXiv  Detail & Related papers  (2023-10-24T23:06:29Z)
• Exploring Open-Vocabulary Semantic Segmentation without Human Labels [76.15862573035565]
  We present ZeroSeg, a novel method that leverages the existing pretrained vision-language model (VL) to train semantic segmentation models. ZeroSeg overcomes this by distilling the visual concepts learned by VL models into a set of segment tokens, each summarizing a localized region of the target image. Our approach achieves state-of-the-art performance when compared to other zero-shot segmentation methods under the same training data.
  arXiv  Detail & Related papers  (2023-06-01T08:47:06Z)
• CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations [90.50864830038202]
  We present Contrastive Spatial Pre-Training (CSP), a self-supervised learning framework for geo-tagged images. We use a dual-encoder to separately encode the images and their corresponding geo-locations, and use contrastive objectives to learn effective location representations from images. CSP significantly boosts the model performance with 10-34% relative improvement with various labeled training data sampling ratios.
  arXiv  Detail & Related papers  (2023-05-01T23:11:18Z)
• Neural Congealing: Aligning Images to a Joint Semantic Atlas [14.348512536556413]
  We present a zero-shot self-supervised framework for aligning semantically-common content across a set of images. Our approach harnesses the power of pre-trained DINO-ViT features to learn. We show that our method performs favorably compared to a state-of-the-art method that requires extensive training on large-scale datasets.
  arXiv  Detail & Related papers  (2023-02-08T09:26:22Z)
• LEAD: Self-Supervised Landmark Estimation by Aligning Distributions of Feature Similarity [49.84167231111667]
  Existing works in self-supervised landmark detection are based on learning dense (pixel-level) feature representations from an image. We introduce an approach to enhance the learning of dense equivariant representations in a self-supervised fashion. We show that having such a prior in the feature extractor helps in landmark detection, even under drastically limited number of annotations.
  arXiv  Detail & Related papers  (2022-04-06T17:48:18Z)
• Learning Contrastive Representation for Semantic Correspondence [150.29135856909477]
  We propose a multi-level contrastive learning approach for semantic matching. We show that image-level contrastive learning is a key component to encourage the convolutional features to find correspondence between similar objects.
  arXiv  Detail & Related papers  (2021-09-22T18:34:14Z)
• Semantic Segmentation with Generative Models: Semi-Supervised Learning and Strong Out-of-Domain Generalization [112.68171734288237]
  We propose a novel framework for discriminative pixel-level tasks using a generative model of both images and labels. We learn a generative adversarial network that captures the joint image-label distribution and is trained efficiently using a large set of unlabeled images. We demonstrate strong in-domain performance compared to several baselines, and are the first to showcase extreme out-of-domain generalization.
  arXiv  Detail & Related papers  (2021-04-12T21:41:25Z)
• Deep Active Learning for Joint Classification & Segmentation with Weak Annotator [22.271760669551817]
  CNN visualization and interpretation methods, like class-activation maps (CAMs), are typically used to highlight the image regions linked to class predictions. We propose an active learning framework, which progressively integrates pixel-level annotations during training. Our results indicate that, by simply using random sample selection, the proposed approach can significantly outperform state-of-the-art CAMs and AL methods.
  arXiv  Detail & Related papers  (2020-10-10T03:25:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.