Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization

• URL: http://arxiv.org/abs/2403.14973v2
• Date: Wed, 7 Aug 2024 08:53:25 GMT
• Title: Pose-Aware Self-Supervised Learning with Viewpoint Trajectory Regularization
• Authors: Jiayun Wang, Yubei Chen, Stella X. Yu,
• Abstract summary: We introduce a new dataset of adjacent image triplets obtained from a viewpoint trajectory. We benchmark both semantic classification and pose estimation accuracies on the same visual feature. Our experiments demonstrate that this approach helps develop a visual representation that encodes object identity.
• Score: 40.5076868823241
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning visual features from unlabeled images has proven successful for semantic categorization, often by mapping different $views$ of the same object to the same feature to achieve recognition invariance. However, visual recognition involves not only identifying $what$ an object is but also understanding $how$ it is presented. For example, seeing a car from the side versus head-on is crucial for deciding whether to stay put or jump out of the way. While unsupervised feature learning for downstream viewpoint reasoning is important, it remains under-explored, partly due to the lack of a standardized evaluation method and benchmarks. We introduce a new dataset of adjacent image triplets obtained from a viewpoint trajectory, without any semantic or pose labels. We benchmark both semantic classification and pose estimation accuracies on the same visual feature. Additionally, we propose a viewpoint trajectory regularization loss for learning features from unlabeled image triplets. Our experiments demonstrate that this approach helps develop a visual representation that encodes object identity and organizes objects by their poses, retaining semantic classification accuracy while achieving emergent global pose awareness and better generalization to novel objects. Our dataset and code are available at http://pwang.pw/trajSSL/.

Related papers

• Measuring the Interpretability of Unsupervised Representations via Quantized Reverse Probing [97.70862116338554]
  We investigate the problem of measuring interpretability of self-supervised representations. We formulate the latter as estimating the mutual information between the representation and a space of manually labelled concepts. We use our method to evaluate a large number of self-supervised representations, ranking them by interpretability.
  arXiv  Detail & Related papers  (2022-09-07T16:18:50Z)
• Matching Multiple Perspectives for Efficient Representation Learning [0.0]
  We present an approach that combines self-supervised learning with a multi-perspective matching technique. We show that the availability of multiple views of the same object combined with a variety of self-supervised pretraining algorithms can lead to improved object classification performance.
  arXiv  Detail & Related papers  (2022-08-16T10:33:13Z)
• Self-Supervised Learning of Object Parts for Semantic Segmentation [7.99536002595393]
  We argue that self-supervised learning of object parts is a solution to this issue. Our method surpasses the state-of-the-art on three semantic segmentation benchmarks by 17%-3%.
  arXiv  Detail & Related papers  (2022-04-27T17:55:17Z)
• Towards Self-Supervised Learning of Global and Object-Centric Representations [4.36572039512405]
  We discuss key aspects of learning structured object-centric representations with self-supervision. We validate our insights through several experiments on the CLEVR dataset.
  arXiv  Detail & Related papers  (2022-03-11T15:18:47Z)
• Unsupervised Part Discovery from Contrastive Reconstruction [90.88501867321573]
  The goal of self-supervised visual representation learning is to learn strong, transferable image representations. We propose an unsupervised approach to object part discovery and segmentation. Our method yields semantic parts consistent across fine-grained but visually distinct categories.
  arXiv  Detail & Related papers  (2021-11-11T17:59:42Z)
• Zero-shot Learning with Deep Neural Networks for Object Recognition [8.572654816871873]
  Zero-shot learning deals with the ability to recognize objects without any visual training sample. This chapter presents a review of the approaches based on deep neural networks to tackle the ZSL problem.
  arXiv  Detail & Related papers  (2021-02-05T12:27:42Z)
• Synthesizing the Unseen for Zero-shot Object Detection [72.38031440014463]
  We propose to synthesize visual features for unseen classes, so that the model learns both seen and unseen objects in the visual domain. We use a novel generative model that uses class-semantics to not only generate the features but also to discriminatively separate them.
  arXiv  Detail & Related papers  (2020-10-19T12:36:11Z)
• Mining Cross-Image Semantics for Weakly Supervised Semantic Segmentation [128.03739769844736]
  Two neural co-attentions are incorporated into the classifier to capture cross-image semantic similarities and differences. In addition to boosting object pattern learning, the co-attention can leverage context from other related images to improve localization map inference. Our algorithm sets new state-of-the-arts on all these settings, demonstrating well its efficacy and generalizability.
  arXiv  Detail & Related papers  (2020-07-03T21:53:46Z)
• Debiased Contrastive Learning [64.98602526764599]
  We develop a debiased contrastive objective that corrects for the sampling of same-label datapoints. Empirically, the proposed objective consistently outperforms the state-of-the-art for representation learning in vision, language, and reinforcement learning benchmarks.
  arXiv  Detail & Related papers  (2020-07-01T04:25:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.