Proportional infinite-width infinite-depth limit for deep linear neural networks

• URL: http://arxiv.org/abs/2411.15267v1
• Date: Fri, 22 Nov 2024 11:25:52 GMT
• Title: Proportional infinite-width infinite-depth limit for deep linear neural networks
• Authors: Federico Bassetti, Lucia Ladelli, Pietro Rotondo,
• Abstract summary: We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. We explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs.
• Score: 0.16385815610837165
• License:
• Abstract: We study the distributional properties of linear neural networks with random parameters in the context of large networks, where the number of layers diverges in proportion to the number of neurons per layer. Prior works have shown that in the infinite-width regime, where the number of neurons per layer grows to infinity while the depth remains fixed, neural networks converge to a Gaussian process, known as the Neural Network Gaussian Process. However, this Gaussian limit sacrifices descriptive power, as it lacks the ability to learn dependent features and produce output correlations that reflect observed labels. Motivated by these limitations, we explore the joint proportional limit in which both depth and width diverge but maintain a constant ratio, yielding a non-Gaussian distribution that retains correlations between outputs. Our contribution extends previous works by rigorously characterizing, for linear activation functions, the limiting distribution as a nontrivial mixture of Gaussians.

Related papers

• Wide Deep Neural Networks with Gaussian Weights are Very Close to Gaussian Processes [1.0878040851638]
  We show that the distance between the network output and the corresponding Gaussian approximation scales inversely with the width of the network, exhibiting faster convergence than the naive suggested by the central limit theorem. We also apply our bounds to obtain theoretical approximations for the exact posterior distribution of the network, when the likelihood is a bounded Lipschitz function of the network output evaluated on a (finite) training set.
  arXiv  Detail & Related papers  (2023-12-18T22:29:40Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Uniform Convergence of Deep Neural Networks with Lipschitz Continuous Activation Functions and Variable Widths [3.0069322256338906]
  We consider deep neural networks with a Lipschitz continuous activation function and with weight matrices of variable widths. In particular, as convolutional neural networks are special deep neural networks with weight matrices of increasing widths, we put forward conditions on the mask sequence. The Lipschitz continuity assumption on the activation functions allows us to include in our theory most of commonly used activation functions in applications.
  arXiv  Detail & Related papers  (2023-06-02T17:07:12Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Infinite-width limit of deep linear neural networks [0.0]
  We study the infinite-width limit of deep linear neural networks with random parameters. We show that the linear predictors converge at an exponential rate to the minimal $ell$-norm minimizer of the risk.
  arXiv  Detail & Related papers  (2022-11-29T18:37:14Z)
• On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks [91.3755431537592]
  We study how random pruning of the weights affects a neural network's neural kernel (NTK) In particular, this work establishes an equivalence of the NTKs between a fully-connected neural network and its randomly pruned version.
  arXiv  Detail & Related papers  (2022-03-27T15:22:19Z)
• Quantitative Gaussian Approximation of Randomly Initialized Deep Neural Networks [1.0878040851638]
  We show how the hidden and output layers sizes affect the Gaussian behaviour of the network. Our explicit inequalities indicate how the hidden and output layers sizes affect the Gaussian behaviour of the network.
  arXiv  Detail & Related papers  (2022-03-14T14:20:19Z)
• Non-asymptotic approximations of neural networks by Gaussian processes [7.56714041729893]
  We study the extent to which wide neural networks may be approximated by Gaussian processes when with random weights. As the width of a network goes to infinity, its law converges to that of a Gaussian process.
  arXiv  Detail & Related papers  (2021-02-17T10:19:26Z)
• Bayesian Deep Ensembles via the Neural Tangent Kernel [49.569912265882124]
  We explore the link between deep ensembles and Gaussian processes (GPs) through the lens of the Neural Tangent Kernel (NTK) We introduce a simple modification to standard deep ensembles training, through addition of a computationally-tractable, randomised and untrainable function to each ensemble member. We prove that our Bayesian deep ensembles make more conservative predictions than standard deep ensembles in the infinite width limit.
  arXiv  Detail & Related papers  (2020-07-11T22:10:52Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• On Random Kernels of Residual Architectures [93.94469470368988]
  We derive finite width and depth corrections for the Neural Tangent Kernel (NTK) of ResNets and DenseNets. Our findings show that in ResNets, convergence to the NTK may occur when depth and width simultaneously tend to infinity. In DenseNets, however, convergence of the NTK to its limit as the width tends to infinity is guaranteed.
  arXiv  Detail & Related papers  (2020-01-28T16:47:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.