Steerable Equivariant Representation Learning
- URL: http://arxiv.org/abs/2302.11349v1
- Date: Wed, 22 Feb 2023 12:42:45 GMT
- Title: Steerable Equivariant Representation Learning
- Authors: Sangnie Bhardwaj, Willie McClinton, Tongzhou Wang, Guillaume Lajoie,
Chen Sun, Phillip Isola, Dilip Krishnan
- Abstract summary: In this paper, we propose a method of learning representations that are instead equivariant to data augmentations.
We demonstrate that our resulting steerable and equivariant representations lead to better performance on transfer learning and robustness.
- Score: 36.138305341173414
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Pre-trained deep image representations are useful for post-training tasks
such as classification through transfer learning, image retrieval, and object
detection. Data augmentations are a crucial aspect of pre-training robust
representations in both supervised and self-supervised settings. Data
augmentations explicitly or implicitly promote invariance in the embedding
space to the input image transformations. This invariance reduces
generalization to those downstream tasks which rely on sensitivity to these
particular data augmentations. In this paper, we propose a method of learning
representations that are instead equivariant to data augmentations. We achieve
this equivariance through the use of steerable representations. Our
representations can be manipulated directly in embedding space via learned
linear maps. We demonstrate that our resulting steerable and equivariant
representations lead to better performance on transfer learning and robustness:
e.g. we improve linear probe top-1 accuracy by between 1% to 3% for transfer;
and ImageNet-C accuracy by upto 3.4%. We further show that the steerability of
our representations provides significant speedup (nearly 50x) for test-time
augmentations; by applying a large number of augmentations for
out-of-distribution detection, we significantly improve OOD AUC on the
ImageNet-C dataset over an invariant representation.
Related papers
- Random Field Augmentations for Self-Supervised Representation Learning [4.3543354293465155]
We propose a new family of local transformations based on Gaussian random fields to generate image augmentations for self-supervised representation learning.
We achieve a 1.7% top-1 accuracy improvement over baseline on ImageNet downstream classification, and a 3.6% improvement on out-of-distribution iNaturalist downstream classification.
While mild transformations improve representations, we observe that strong transformations can degrade the structure of an image.
arXiv Detail & Related papers (2023-11-07T00:35:09Z) - SPOT: Scalable 3D Pre-training via Occupancy Prediction for Learning Transferable 3D Representations [76.45009891152178]
Pretraining-finetuning approach can alleviate the labeling burden by fine-tuning a pre-trained backbone across various downstream datasets as well as tasks.
We show, for the first time, that general representations learning can be achieved through the task of occupancy prediction.
Our findings will facilitate the understanding of LiDAR points and pave the way for future advancements in LiDAR pre-training.
arXiv Detail & Related papers (2023-09-19T11:13:01Z) - Effective Data Augmentation With Diffusion Models [65.09758931804478]
We address the lack of diversity in data augmentation with image-to-image transformations parameterized by pre-trained text-to-image diffusion models.
Our method edits images to change their semantics using an off-the-shelf diffusion model, and generalizes to novel visual concepts from a few labelled examples.
We evaluate our approach on few-shot image classification tasks, and on a real-world weed recognition task, and observe an improvement in accuracy in tested domains.
arXiv Detail & Related papers (2023-02-07T20:42:28Z) - CIPER: Combining Invariant and Equivariant Representations Using
Contrastive and Predictive Learning [6.117084972237769]
We introduce Contrastive Invariant and Predictive Equivariant Representation learning (CIPER)
CIPER comprises both invariant and equivariant learning objectives using one shared encoder and two different output heads on top of the encoder.
We evaluate our method on static image tasks and time-augmented image datasets.
arXiv Detail & Related papers (2023-02-05T07:50:46Z) - Soft Augmentation for Image Classification [68.71067594724663]
We propose generalizing augmentation with invariant transforms to soft augmentation.
We show that soft targets allow for more aggressive data augmentation.
We also show that soft augmentations generalize to self-supervised classification tasks.
arXiv Detail & Related papers (2022-11-09T01:04:06Z) - Improving Transferability of Representations via Augmentation-Aware
Self-Supervision [117.15012005163322]
AugSelf is an auxiliary self-supervised loss that learns the difference of augmentation parameters between two randomly augmented samples.
Our intuition is that AugSelf encourages to preserve augmentation-aware information in learned representations, which could be beneficial for their transferability.
AugSelf can easily be incorporated into recent state-of-the-art representation learning methods with a negligible additional training cost.
arXiv Detail & Related papers (2021-11-18T10:43:50Z) - Self-Supervised 3D Hand Pose Estimation from monocular RGB via
Contrastive Learning [50.007445752513625]
We propose a new self-supervised method for the structured regression task of 3D hand pose estimation.
We experimentally investigate the impact of invariant and equivariant contrastive objectives.
We show that a standard ResNet-152, trained on additional unlabeled data, attains an improvement of $7.6%$ in PA-EPE on FreiHAND.
arXiv Detail & Related papers (2021-06-10T17:48:57Z) - InAugment: Improving Classifiers via Internal Augmentation [14.281619356571724]
We present a novel augmentation operation, that exploits image internal statistics.
We show improvement over state-of-the-art augmentation techniques.
We also demonstrate an increase for ResNet50 and EfficientNet-B3 top-1's accuracy on the ImageNet dataset.
arXiv Detail & Related papers (2021-04-08T15:37:21Z) - On the Generalization Effects of Linear Transformations in Data
Augmentation [32.01435459892255]
Data augmentation is a powerful technique to improve performance in applications such as image and text classification tasks.
We study a family of linear transformations and study their effects on the ridge estimator in an over-parametrized linear regression setting.
We propose an augmentation scheme that searches over the space of transformations by how uncertain the model is about the transformed data.
arXiv Detail & Related papers (2020-05-02T04:10:21Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.