Dissecting Hessian: Understanding Common Structure of Hessian in Neural
Networks
- URL: http://arxiv.org/abs/2010.04261v5
- Date: Wed, 16 Jun 2021 15:27:49 GMT
- Title: Dissecting Hessian: Understanding Common Structure of Hessian in Neural
Networks
- Authors: Yikai Wu, Xingyu Zhu, Chenwei Wu, Annie Wang, Rong Ge
- Abstract summary: Hessian captures important properties of the deep neural network loss landscape.
We make new observations about the top eigenspace of layer-wise Hessian.
We show that the new eigenspace structure can be explained by approximating the Hessian using Kronecker factorization.
- Score: 11.57132149295061
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Hessian captures important properties of the deep neural network loss
landscape. Previous works have observed low rank structure in the Hessians of
neural networks. We make several new observations about the top eigenspace of
layer-wise Hessian: top eigenspaces for different models have surprisingly high
overlap, and top eigenvectors form low rank matrices when they are reshaped
into the same shape as the corresponding weight matrix. Towards formally
explaining such structures of the Hessian, we show that the new eigenspace
structure can be explained by approximating the Hessian using Kronecker
factorization; we also prove the low rank structure for random data at random
initialization for over-parametrized two-layer neural nets. Our new
understanding can explain why some of these structures become weaker when the
network is trained with batch normalization. The Kronecker factorization also
leads to better explicit generalization bounds.
Related papers
- Hessian Eigenvectors and Principal Component Analysis of Neural Network
Weight Matrices [0.0]
This study delves into the intricate dynamics of trained deep neural networks and their relationships with network parameters.
We unveil a correlation between Hessian eigenvectors and network weights.
This relationship, hinging on the magnitude of eigenvalues, allows us to discern parameter directions within the network.
arXiv Detail & Related papers (2023-11-01T11:38:31Z) - The Hessian perspective into the Nature of Convolutional Neural Networks [32.7270996241955]
We develop a framework relying on Toeplitz representation of CNNs, and then utilize it to reveal the Hessian structure and, in particular, its rank.
Overall, our work generalizes and establishes the key insight that, even in CNNs, the Hessian rank grows as the square root of the number of parameters.
arXiv Detail & Related papers (2023-05-16T01:15:00Z) - Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights.
We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
arXiv Detail & Related papers (2023-02-01T03:18:07Z) - Dynamic Inference with Neural Interpreters [72.90231306252007]
We present Neural Interpreters, an architecture that factorizes inference in a self-attention network as a system of modules.
inputs to the model are routed through a sequence of functions in a way that is end-to-end learned.
We show that Neural Interpreters perform on par with the vision transformer using fewer parameters, while being transferrable to a new task in a sample efficient manner.
arXiv Detail & Related papers (2021-10-12T23:22:45Z) - The Separation Capacity of Random Neural Networks [78.25060223808936]
We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability.
We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
arXiv Detail & Related papers (2021-07-31T10:25:26Z) - Analytic Insights into Structure and Rank of Neural Network Hessian Maps [32.90143789616052]
Hessian of a neural network captures parameter interactions through second-order derivatives of the loss.
We develop theoretical tools to analyze the range of the Hessian map, providing us with a precise understanding of its rank deficiency.
This yields exact formulas and tight upper bounds for the Hessian rank of deep linear networks.
arXiv Detail & Related papers (2021-06-30T17:29:58Z) - Redundant representations help generalization in wide neural networks [71.38860635025907]
We study the last hidden layer representations of various state-of-the-art convolutional neural networks.
We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
arXiv Detail & Related papers (2021-06-07T10:18:54Z) - Dual-constrained Deep Semi-Supervised Coupled Factorization Network with
Enriched Prior [80.5637175255349]
We propose a new enriched prior based Dual-constrained Deep Semi-Supervised Coupled Factorization Network, called DS2CF-Net.
To ex-tract hidden deep features, DS2CF-Net is modeled as a deep-structure and geometrical structure-constrained neural network.
Our network can obtain state-of-the-art performance for representation learning and clustering.
arXiv Detail & Related papers (2020-09-08T13:10:21Z) - Revealing the Structure of Deep Neural Networks via Convex Duality [70.15611146583068]
We study regularized deep neural networks (DNNs) and introduce a convex analytic framework to characterize the structure of hidden layers.
We show that a set of optimal hidden layer weights for a norm regularized training problem can be explicitly found as the extreme points of a convex set.
We apply the same characterization to deep ReLU networks with whitened data and prove the same weight alignment holds.
arXiv Detail & Related papers (2020-02-22T21:13:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.