Convergence of Deep Convolutional Neural Networks
- URL: http://arxiv.org/abs/2109.13542v1
- Date: Tue, 28 Sep 2021 07:48:17 GMT
- Title: Convergence of Deep Convolutional Neural Networks
- Authors: Yuesheng Xu and Haizhang Zhang
- Abstract summary: Convergence of deep neural networks as the depth of the networks tends to infinity is fundamental in building the mathematical foundation for deep learning.
We first study convergence of general ReLU networks with increasing widths and then apply the results obtained to deep convolutional neural networks.
- Score: 2.5991265608180396
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Convergence of deep neural networks as the depth of the networks tends to
infinity is fundamental in building the mathematical foundation for deep
learning. In a previous study, we investigated this question for deep ReLU
networks with a fixed width. This does not cover the important convolutional
neural networks where the widths are increasing from layer to layer. For this
reason, we first study convergence of general ReLU networks with increasing
widths and then apply the results obtained to deep convolutional neural
networks. It turns out the convergence reduces to convergence of infinite
products of matrices with increasing sizes, which has not been considered in
the literature. We establish sufficient conditions for convergence of such
infinite products of matrices. Based on the conditions, we present sufficient
conditions for piecewise convergence of general deep ReLU networks with
increasing widths, and as well as pointwise convergence of deep ReLU
convolutional neural networks.
Related papers
- Feature Learning and Generalization in Deep Networks with Orthogonal Weights [1.7956122940209063]
Deep neural networks with numerically weights from independent Gaussian distributions can be tuned to criticality.
These networks still exhibit fluctuations that grow linearly with the depth of the network.
We show analytically that rectangular networks with tanh activations and weights from the ensemble of matrices have corresponding preactivation fluctuations.
arXiv Detail & Related papers (2023-10-11T18:00:02Z) - Depth Degeneracy in Neural Networks: Vanishing Angles in Fully Connected ReLU Networks on Initialization [5.678271181959529]
We study the evolution of the angle between two inputs to a ReLU neural network as a function of the number of layers.
We validate our theoretical results with Monte Carlo experiments and show that our results accurately approximate finite network behaviour.
We also empirically investigate how the depth degeneracy phenomenon can negatively impact training of real networks.
arXiv Detail & Related papers (2023-02-20T01:30:27Z) - Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK [86.45209429863858]
We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime.
We show that the neural networks possess a different limiting kernel which we call textitbias-generalized NTK
We also study various properties of the neural networks with this new kernel.
arXiv Detail & Related papers (2023-01-01T02:11:39Z) - Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory.
We show that linear networks make provably optimal predictions at infinite depth.
We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
arXiv Detail & Related papers (2022-12-29T20:57:46Z) - Convergence of Deep Neural Networks with General Activation Functions
and Pooling [5.316908050163474]
Convergence of deep neural networks is a fundamental issue in building the mathematical foundation for deep learning.
We study the convergence of deep neural networks as the depth tends to infinity for two other activation functions: the leaky ReLU and the sigmoid function.
arXiv Detail & Related papers (2022-05-13T11:49:03Z) - Path Regularization: A Convexity and Sparsity Inducing Regularization
for Parallel ReLU Networks [75.33431791218302]
We study the training problem of deep neural networks and introduce an analytic approach to unveil hidden convexity in the optimization landscape.
We consider a deep parallel ReLU network architecture, which also includes standard deep networks and ResNets as its special cases.
arXiv Detail & Related papers (2021-10-18T18:00:36Z) - Convergence of Deep ReLU Networks [2.5991265608180396]
We explore convergence of deep neural networks with the popular ReLU activation function.
We identify the convergence of the ReLU networks as convergence of a class of infinite products of matrices.
These results provide mathematical insights to the design strategy of the well-known deep residual networks in image classification.
arXiv Detail & Related papers (2021-07-27T00:33:53Z) - A Convergence Theory Towards Practical Over-parameterized Deep Neural
Networks [56.084798078072396]
We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time.
We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both.
Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
arXiv Detail & Related papers (2021-01-12T00:40:45Z) - Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
kernel methods outperform fully-connected finite-width networks.
Centered and ensembled finite networks have reduced posterior variance.
Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
arXiv Detail & Related papers (2020-07-31T01:57:47Z) - On Random Kernels of Residual Architectures [93.94469470368988]
We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets.
Our findings show that in ResNets, convergence to the NTK may occur when depth and width simultaneously tend to infinity.
In DenseNets, however, convergence of the NTK to its limit as the width tends to infinity is guaranteed.
arXiv Detail & Related papers (2020-01-28T16:47:53Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.