Critical initialization destabilizes higher input derivatives in wide scalar-input networks

• URL: http://arxiv.org/abs/2609.09244v1
• Date: Tue, 08 Sep 2026 10:17:45 GMT
• Title: Critical initialization destabilizes higher input derivatives in wide scalar-input networks
• Abstract summary: We prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions.<n>For residual networks with branch scale L-1/2, we prove that every fixed finite derivative order has uniformly bounded variance.
• Score: 3.0318216701273397
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The edge-of-chaos condition preserves first-order input perturbations in wide randomly initialized networks, but physics-informed losses, score matching and derivative regularization depend on higher input derivatives. For smooth scalar-input fully connected networks, using a joint Gaussianity of the finite derivative jet that holds in the infinite-width limit at each fixed depth, we derive mean-field recursions through third order that are exact at the variance fixed point, with finite-depth corrections that decay geometrically. At criticality, the first-derivative variance is depth-invariant, whereas the second-derivative variance grows linearly whenever the activation has nonzero curvature. The resulting third-order system closes on mean-field susceptibilities. For residual networks with branch scale L^{-1/2}, we prove that every fixed finite derivative order has uniformly bounded variance under explicit regularity assumptions. Simulations verify the critical growth laws, the residual bound, and the closed recursion. The results concern initialization, not trained-network performance.

Related papers

• Branch Geometry and Finite-Radius Sensitivity of Hard-ReLU Training [2.969260589680338]
  Outer-learning algorithms use infinitesimal sensitivities to propose finite changes to training parameters.<n>For hard-ReLU training, the derivative of the finite program and the derivative of its flow limit do not by themselves specify the response at a chosen radius.<n>We characterize the intervening regime in which the radius is proportional to the GD step event rounding then survives at leading order.
  arXiv  Detail & Related papers  (2026-08-31T15:26:59Z)
• Optimal Non-Asymptotic Edgeworth Expansions for Multivariate Neural Network Outputs [0.0]
  We approximate deviations for a neural network evaluated in a finite number of inputs.<n>As an application, we quantify error in Bayesian posterior distributions when the prior is replaced by its Edgeworth expansion.
  arXiv  Detail & Related papers  (2026-05-22T12:45:17Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• A Local Polyak-Lojasiewicz and Descent Lemma of Gradient Descent For Overparametrized Linear Models [6.734175048463699]
  We derive a linear convergence rate for gradient descent for two-layer linear neural networks trained with squared loss.<n>Our convergence analysis not only improves upon prior results but also suggests a better choice for the step size.
  arXiv  Detail & Related papers  (2025-05-16T19:57:22Z)
• Implicit Bias of Gradient Descent for Non-Homogeneous Deep Networks [39.38604477897272]
  We establish the implicit bias of gradient descent (GD) for generic non-homogeneous deep networks under exponential loss.<n>Our results apply to a broad class of non-homogeneous networks satisfying a mild near-homogeneity condition.
  arXiv  Detail & Related papers  (2025-02-22T04:40:45Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks [53.95175206863992]
  We study the type of solutions to which gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. We prove that although shallow ReLU networks are universal approximators, stable shallow networks are not.
  arXiv  Detail & Related papers  (2023-06-30T09:17:39Z)
• Neural Networks with Sparse Activation Induced by Large Bias: Tighter Analysis with Bias-Generalized NTK [86.45209429863858]
  We study training one-hidden-layer ReLU networks in the neural tangent kernel (NTK) regime. We show that the neural networks possess a different limiting kernel which we call textitbias-generalized NTK We also study various properties of the neural networks with this new kernel.
  arXiv  Detail & Related papers  (2023-01-01T02:11:39Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• A Dynamical Central Limit Theorem for Shallow Neural Networks [48.66103132697071]
  We prove that the fluctuations around the mean limit remain bounded in mean square throughout training. If the mean-field dynamics converges to a measure that interpolates the training data, we prove that the deviation eventually vanishes in the CLT scaling.
  arXiv  Detail & Related papers  (2020-08-21T18:00:50Z)
• Implicit Bias of Gradient Descent for Mean Squared Error Regression with Two-Layer Wide Neural Networks [1.3706331473063877]
  We show that the solution of training a width-$n$ shallow ReLU network is within $n- 1/2$ of the function which fits the training data. We also show that the training trajectories are captured by trajectories of smoothing splines with decreasing regularization strength.
  arXiv  Detail & Related papers  (2020-06-12T17:46:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.