Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- URL: http://arxiv.org/abs/2103.03230v1
- Date: Thu, 4 Mar 2021 18:55:09 GMT
- Title: Barlow Twins: Self-Supervised Learning via Redundancy Reduction
- Authors: Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, St\'ephane Deny
- Abstract summary: Self-supervised learning (SSL) is rapidly closing the gap with supervised methods on large computer vision benchmarks.
We propose an objective function that naturally avoids collapse by measuring the cross-correlation matrix between the outputs of two identical networks.
This causes the representation vectors of distorted versions of a sample to be similar, while minimizing the redundancy between the components of these vectors.
- Score: 31.077182488826963
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Self-supervised learning (SSL) is rapidly closing the gap with supervised
methods on large computer vision benchmarks. A successful approach to SSL is to
learn representations which are invariant to distortions of the input sample.
However, a recurring issue with this approach is the existence of trivial
constant representations. Most current methods avoid such collapsed solutions
by careful implementation details. We propose an objective function that
naturally avoids such collapse by measuring the cross-correlation matrix
between the outputs of two identical networks fed with distorted versions of a
sample, and making it as close to the identity matrix as possible. This causes
the representation vectors of distorted versions of a sample to be similar,
while minimizing the redundancy between the components of these vectors. The
method is called Barlow Twins, owing to neuroscientist H. Barlow's
redundancy-reduction principle applied to a pair of identical networks. Barlow
Twins does not require large batches nor asymmetry between the network twins
such as a predictor network, gradient stopping, or a moving average on the
weight updates. It allows the use of very high-dimensional output vectors.
Barlow Twins outperforms previous methods on ImageNet for semi-supervised
classification in the low-data regime, and is on par with current state of the
art for ImageNet classification with a linear classifier head, and for transfer
tasks of classification and object detection.
Related papers
- Unsupervised Representation Learning by Balanced Self Attention Matching [2.3020018305241337]
We present a self-supervised method for embedding image features called BAM.
We obtain rich representations and avoid feature collapse by minimizing a loss that matches these distributions to their globally balanced and entropy regularized version.
We show competitive performance with leading methods on both semi-supervised and transfer-learning benchmarks.
arXiv Detail & Related papers (2024-08-04T12:52:44Z) - Guarding Barlow Twins Against Overfitting with Mixed Samples [27.7244906436942]
Self-supervised learning aims to learn transferable feature representations for downstream applications without relying on labeled data.
We introduce Mixed Barlow Twins, which aims to improve sample interaction during Barlow Twins training via linearly interpolated samples.
arXiv Detail & Related papers (2023-12-04T18:59:36Z) - Boosting Few-shot Fine-grained Recognition with Background Suppression
and Foreground Alignment [53.401889855278704]
Few-shot fine-grained recognition (FS-FGR) aims to recognize novel fine-grained categories with the help of limited available samples.
We propose a two-stage background suppression and foreground alignment framework, which is composed of a background activation suppression (BAS) module, a foreground object alignment (FOA) module, and a local to local (L2L) similarity metric.
Experiments conducted on multiple popular fine-grained benchmarks demonstrate that our method outperforms the existing state-of-the-art by a large margin.
arXiv Detail & Related papers (2022-10-04T07:54:40Z) - Non-contrastive representation learning for intervals from well logs [58.70164460091879]
The representation learning problem in the oil & gas industry aims to construct a model that provides a representation based on logging data for a well interval.
One of the possible approaches is self-supervised learning (SSL)
We are the first to introduce non-contrastive SSL for well-logging data.
arXiv Detail & Related papers (2022-09-28T13:27:10Z) - Large-Margin Representation Learning for Texture Classification [67.94823375350433]
This paper presents a novel approach combining convolutional layers (CLs) and large-margin metric learning for training supervised models on small datasets for texture classification.
The experimental results on texture and histopathologic image datasets have shown that the proposed approach achieves competitive accuracy with lower computational cost and faster convergence when compared to equivalent CNNs.
arXiv Detail & Related papers (2022-06-17T04:07:45Z) - Interpolation-based Correlation Reduction Network for Semi-Supervised
Graph Learning [49.94816548023729]
We propose a novel graph contrastive learning method, termed Interpolation-based Correlation Reduction Network (ICRN)
In our method, we improve the discriminative capability of the latent feature by enlarging the margin of decision boundaries.
By combining the two settings, we extract rich supervision information from both the abundant unlabeled nodes and the rare yet valuable labeled nodes for discnative representation learning.
arXiv Detail & Related papers (2022-06-06T14:26:34Z) - Compare learning: bi-attention network for few-shot learning [6.559037166322981]
One of the Few-shot learning methods called metric learning addresses this challenge by first learning a deep distance metric to determine whether a pair of images belong to the same category.
In this paper, we propose a novel approach named Bi-attention network to compare the instances, which can measure the similarity between embeddings of instances precisely, globally and efficiently.
arXiv Detail & Related papers (2022-03-25T07:39:10Z) - Barlow Graph Auto-Encoder for Unsupervised Network Embedding [6.900303913555705]
We propose Barlow Graph Auto-Encoder, a simple yet effective architecture for learning network embedding.
It aims to maximize the similarity between the embedding vectors of immediate and larger neighborhoods of a node, while minimizing the redundancy between the components of these projections.
Our approach yields promising results for inductive link prediction and is also on par with state of the art for clustering and downstream node classification.
arXiv Detail & Related papers (2021-10-29T12:30:43Z) - Self-Supervised Classification Network [3.8073142980733]
Self-supervised end-to-end classification neural network learns labels and representations simultaneously.
First unsupervised end-to-end classification network to perform well on the large-scale ImageNet dataset.
arXiv Detail & Related papers (2021-03-19T19:29:42Z) - Unsupervised Person Re-identification via Softened Similarity Learning [122.70472387837542]
Person re-identification (re-ID) is an important topic in computer vision.
This paper studies the unsupervised setting of re-ID, which does not require any labeled information.
Experiments on two image-based and video-based datasets demonstrate state-of-the-art performance.
arXiv Detail & Related papers (2020-04-07T17:16:41Z) - Embedding Propagation: Smoother Manifold for Few-Shot Classification [131.81692677836202]
We propose to use embedding propagation as an unsupervised non-parametric regularizer for manifold smoothing in few-shot classification.
We empirically show that embedding propagation yields a smoother embedding manifold.
We show that embedding propagation consistently improves the accuracy of the models in multiple semi-supervised learning scenarios by up to 16% points.
arXiv Detail & Related papers (2020-03-09T13:51:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.