The Mechanism of Prediction Head in Non-contrastive Self-supervised
Learning
- URL: http://arxiv.org/abs/2205.06226v2
- Date: Sat, 14 May 2022 05:04:42 GMT
- Title: The Mechanism of Prediction Head in Non-contrastive Self-supervised
Learning
- Authors: Zixin Wen, Yuanzhi Li
- Abstract summary: We present our empirical and theoretical discoveries on non-contrastive self-supervised learning.
We find that when the prediction head is as an identity matrix with only its off-diagonal entries being trainable, the network can learn competitive representations.
This is the first end-to-end optimization guarantee for non-contrastive methods using nonlinear neural networks.
- Score: 38.71821080323513
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recently the surprising discovery of the Bootstrap Your Own Latent (BYOL)
method by Grill et al. shows the negative term in contrastive loss can be
removed if we add the so-called prediction head to the network. This initiated
the research of non-contrastive self-supervised learning. It is mysterious why
even when there exist trivial collapsed global optimal solutions, neural
networks trained by (stochastic) gradient descent can still learn competitive
representations. This phenomenon is a typical example of implicit bias in deep
learning and remains little understood.
In this work, we present our empirical and theoretical discoveries on
non-contrastive self-supervised learning. Empirically, we find that when the
prediction head is initialized as an identity matrix with only its off-diagonal
entries being trainable, the network can learn competitive representations even
though the trivial optima still exist in the training objective. Theoretically,
we present a framework to understand the behavior of the trainable, but
identity-initialized prediction head. Under a simple setting, we characterized
the substitution effect and acceleration effect of the prediction head. The
substitution effect happens when learning the stronger features in some neurons
can substitute for learning these features in other neurons through updating
the prediction head. And the acceleration effect happens when the substituted
features can accelerate the learning of other weaker features to prevent them
from being ignored. These two effects enable the neural networks to learn all
the features rather than focus only on learning the stronger features, which is
likely the cause of the dimensional collapse phenomenon. To the best of our
knowledge, this is also the first end-to-end optimization guarantee for
non-contrastive methods using nonlinear neural networks with a trainable
prediction head and normalization.
Related papers
- Instance-wise Linearization of Neural Network for Model Interpretation [13.583425552511704]
The challenge can dive into the non-linear behavior of the neural network.
For a neural network model, the non-linear behavior is often caused by non-linear activation units of a model.
We propose an instance-wise linearization approach to reformulates the forward computation process of a neural network prediction.
arXiv Detail & Related papers (2023-10-25T02:07:39Z) - Grokking as the Transition from Lazy to Rich Training Dynamics [35.186196991224286]
grokking occurs when the train loss of a neural network decreases much earlier than its test loss.
Key determinants of grokking are the rate of feature learning and the alignment of the initial features with the target function.
arXiv Detail & Related papers (2023-10-09T19:33:21Z) - Utility-Probability Duality of Neural Networks [4.871730595406078]
We propose an alternative utility-based explanation to the standard supervised learning procedure in deep learning.
The basic idea is to interpret the learned neural network not as a probability model but as an ordinal utility function.
We show that for all neural networks with softmax outputs, the SGD learning dynamic of maximum likelihood estimation can be seen as an iteration process.
arXiv Detail & Related papers (2023-05-24T08:09:07Z) - Theoretical Characterization of How Neural Network Pruning Affects its
Generalization [131.1347309639727]
This work makes the first attempt to study how different pruning fractions affect the model's gradient descent dynamics and generalization.
It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero.
More surprisingly, the generalization bound gets better as the pruning fraction gets larger.
arXiv Detail & Related papers (2023-01-01T03:10:45Z) - Predictive Coding: Towards a Future of Deep Learning beyond
Backpropagation? [41.58529335439799]
The backpropagation of error algorithm used to train deep neural networks has been fundamental to the successes of deep learning.
Recent work has developed the idea into a general-purpose algorithm able to train neural networks using only local computations.
We show the substantially greater flexibility of predictive coding networks against equivalent deep neural networks.
arXiv Detail & Related papers (2022-02-18T22:57:03Z) - How does unlabeled data improve generalization in self-training? A
one-hidden-layer theoretical analysis [93.37576644429578]
This work establishes the first theoretical analysis for the known iterative self-training paradigm.
We prove the benefits of unlabeled data in both training convergence and generalization ability.
Experiments from shallow neural networks to deep neural networks are also provided to justify the correctness of our established theoretical insights on self-training.
arXiv Detail & Related papers (2022-01-21T02:16:52Z) - Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task.
This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
arXiv Detail & Related papers (2020-11-18T18:52:08Z) - Vulnerability Under Adversarial Machine Learning: Bias or Variance? [77.30759061082085]
We investigate the effect of adversarial machine learning on the bias and variance of a trained deep neural network.
Our analysis sheds light on why the deep neural networks have poor performance under adversarial perturbation.
We introduce a new adversarial machine learning algorithm with lower computational complexity than well-known adversarial machine learning strategies.
arXiv Detail & Related papers (2020-08-01T00:58:54Z) - Learning from Failure: Training Debiased Classifier from Biased
Classifier [76.52804102765931]
We show that neural networks learn to rely on spurious correlation only when it is "easier" to learn than the desired knowledge.
We propose a failure-based debiasing scheme by training a pair of neural networks simultaneously.
Our method significantly improves the training of the network against various types of biases in both synthetic and real-world datasets.
arXiv Detail & Related papers (2020-07-06T07:20:29Z) - Bidirectionally Self-Normalizing Neural Networks [46.20979546004718]
We provide a rigorous result that shows, under mild conditions, how the vanishing/exploding gradients problem disappears with high probability if the neural networks have sufficient width.
Our main idea is to constrain both forward and backward signal propagation in a nonlinear neural network through a new class of activation functions.
arXiv Detail & Related papers (2020-06-22T12:07:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.