Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods

• URL: http://arxiv.org/abs/2606.06772v1
• Date: Thu, 04 Jun 2026 23:31:52 GMT
• Title: Generalization in Deep Neural Networks: Minimax Rates for Gradient Methods
• Abstract summary: This paper provides a comprehensive generalization analysis of deep neural networks (DNNs) trained using gradient-based methods.<n>We establish, for the first time, a crucial connection between the learning dynamics of a DNN with smooth activation functions trained via gradient-based methods and those of kernel methods.<n>We derive the first known minimax-optimal rates for the excess population risk of gradient descent (GD) and gradient descent (SGD)
• Score: 42.41448048632761
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Understanding the generalization performance of over-parameterized neural networks has become a central topic in deep learning theory. While recent advances, particularly works under the Neural Tangent Kernel (NTK) regime, have shed light on the behavior of shallow architectures, the statistical generalization properties of deep neural networks (DNNs), especially in regression tasks, remain far less understood. In this paper, we make significant progress toward closing this gap by providing a comprehensive generalization analysis of DNNs trained using gradient-based methods. First, we establish, for the first time, a crucial connection between the learning dynamics of a DNN with smooth activation functions trained via gradient-based methods and those of kernel methods, showing that gradient-based methods on over-parameterized DNNs can fully inherit the favorable learning dynamics of their kernel counterparts. Building on this connection and the well-established optimality of kernel methods, we derive the first known minimax-optimal rates for the excess population risk of both gradient descent (GD) and stochastic gradient descent (SGD), under the assumption that network width scales polynomially with the sample size. Our results demonstrate that, with sufficient width, DNNs trained by GD or SGD can achieve generalization performance comparable to kernel-based methods.

Related papers

• Optimal Rates for Generalization of Gradient Descent Methods with Deep Neural Networks [35.78666449629947]
  This paper presents a comprehensive analysis for deep ReLU networks trained using gradient descent (GD) and gradient descent (SGD)<n>We establish the first known minimax-optimal rates of excess population for both GD and SGD with deep ReLU networks.<n>Our results demonstrate that with sufficient width, gradient descent methods for deep ReLU networks can achieve optimal generalization rates on par with kernel methods.
  arXiv  Detail & Related papers  (2026-06-04T23:04:49Z)
• Depth-induced NTK: Bridging Over-parameterized Neural Networks and Deep Neural Kernels [13.302913618949468]
  We provide a principled framework to interpret over- parameterized neural networks by mapping hierarchical feature transformations into kernel spaces.<n>We propose a depth-induced NTK kernel based on a shortcut-related architecture, which converges to a Gaussian process as the network depth approaches infinity.<n>Our findings significantly extend the existing landscape of the neural kernel theory and provide an in-depth understanding of deep learning and the scaling law.
  arXiv  Detail & Related papers  (2025-11-05T10:00:03Z)
• Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel [55.82768375605861]
  We establish a generalization bound for gradient flow that aligns with the classical Rademacher complexity for kernel methods.<n>Unlike static kernels such as NTK, the LPK captures the entire training trajectory, adapting to both data and optimization dynamics.
  arXiv  Detail & Related papers  (2025-06-12T23:17:09Z)
• From Deep Additive Kernel Learning to Last-Layer Bayesian Neural Networks via Induced Prior Approximation [11.917792144592056]
  We propose the Deep Additive Kernel (DAK) model, which incorporates an additive structure for the last-layer GP.<n>The proposed method enjoys the interpretability of DKL as well as the computational advantages of BNN.<n> Empirical results show that the proposed approach outperforms state-of-the-art DKL methods in both regression and classification tasks.
  arXiv  Detail & Related papers  (2025-02-14T20:14:17Z)
• Stochastic Gradient Descent for Two-layer Neural Networks [2.0349026069285423]
  This paper presents a study on the convergence rates of the descent (SGD) algorithm when applied to overparameterized two-layer neural networks. Our approach combines the Tangent Kernel (NTK) approximation with convergence analysis in the Reproducing Kernel Space (RKHS) generated by NTK. Our research framework enables us to explore the intricate interplay between kernel methods and optimization processes, shedding light on the dynamics and convergence properties of neural networks.
  arXiv  Detail & Related papers  (2024-07-10T13:58:57Z)
• Implicit Stochastic Gradient Descent for Training Physics-informed Neural Networks [51.92362217307946]
  Physics-informed neural networks (PINNs) have effectively been demonstrated in solving forward and inverse differential equation problems. PINNs are trapped in training failures when the target functions to be approximated exhibit high-frequency or multi-scale features. In this paper, we propose to employ implicit gradient descent (ISGD) method to train PINNs for improving the stability of training process.
  arXiv  Detail & Related papers  (2023-03-03T08:17:47Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Provable Acceleration of Nesterov's Accelerated Gradient Method over Heavy Ball Method in Training Over-Parameterized Neural Networks [12.475834086073734]
  First-order gradient method has been extensively employed in training neural networks. Recent research has proved that the first neural-order method is capable of attaining a global minimum convergence.
  arXiv  Detail & Related papers  (2022-08-08T07:13:26Z)
• On Feature Learning in Neural Networks with Global Convergence Guarantees [49.870593940818715]
  We study the optimization of wide neural networks (NNs) via gradient flow (GF) We show that when the input dimension is no less than the size of the training set, the training loss converges to zero at a linear rate under GF. We also show empirically that, unlike in the Neural Tangent Kernel (NTK) regime, our multi-layer model exhibits feature learning and can achieve better generalization performance than its NTK counterpart.
  arXiv  Detail & Related papers  (2022-04-22T15:56:43Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.