Generalization Bounds for Rank-sparse Neural Networks

• URL: http://arxiv.org/abs/2510.21945v1
• Date: Fri, 24 Oct 2025 18:19:32 GMT
• Title: Generalization Bounds for Rank-sparse Neural Networks
• Abstract summary: We prove generalization bounds for neural networks which exploit the approximate low rank structure of the weight matrices if present.<n>As $p$ increases, the bound behaves more like a norm-based bound instead.
• Score: 40.956710248523294
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: It has been recently observed in much of the literature that neural networks exhibit a bottleneck rank property: for larger depths, the activation and weights of neural networks trained with gradient-based methods tend to be of approximately low rank. In fact, the rank of the activations of each layer converges to a fixed value referred to as the ``bottleneck rank'', which is the minimum rank required to represent the training data. This perspective is in line with the observation that regularizing linear networks (without activations) with weight decay is equivalent to minimizing the Schatten $p$ quasi norm of the neural network. In this paper we investigate the implications of this phenomenon for generalization. More specifically, we prove generalization bounds for neural networks which exploit the approximate low rank structure of the weight matrices if present. The final results rely on the Schatten $p$ quasi norms of the weight matrices: for small $p$, the bounds exhibit a sample complexity $ \widetilde{O}(WrL^2)$ where $W$ and $L$ are the width and depth of the neural network respectively and where $r$ is the rank of the weight matrices. As $p$ increases, the bound behaves more like a norm-based bound instead.

Related papers

• Neural Networks Learn Generic Multi-Index Models Near Information-Theoretic Limit [66.20349460098275]
  We study the gradient descent learning of a general Gaussian Multi-index model $f(boldsymbolx)=g(boldsymbolUboldsymbolx)$ with hidden subspace $boldsymbolUin mathbbRrtimes d$.<n>We prove that under generic non-degenerate assumptions on the link function, a standard two-layer neural network trained via layer-wise gradient descent can agnostically learn the target with $o_d(1)$ test error.
  arXiv  Detail & Related papers  (2025-11-19T04:46:47Z)
• Low-rank bias, weight decay, and model merging in neural networks [12.352761060862072]
  We show several properties of such deep neural networks, induced by $L2$ regularization.<n>In addition, we investigate a multitask learning phenomenon enabled by $L2$ regularization and low-rank bias.
  arXiv  Detail & Related papers  (2025-02-24T17:17:00Z)
• Weight decay induces low-rank attention layers [8.746774382296008]
  We study the influence of weight decay as well as $L2$-regularization when training neural network models. We verify that the key-query and value-projection matrix products $W_KTW_Q, PW_V$ within attention layers, when optimized with weight decay, indeed induce a significant reduction in the rank.
  arXiv  Detail & Related papers  (2024-10-31T11:04:07Z)
• Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Data [66.1211659120882]
  The implicit bias towards solutions with favorable properties is believed to be a key reason why neural networks trained by gradient-based optimization can generalize well. While the implicit bias of gradient flow has been widely studied for homogeneous neural networks (including ReLU and leaky ReLU networks), the implicit bias of gradient descent is currently only understood for smooth neural networks.
  arXiv  Detail & Related papers  (2023-10-29T08:47:48Z)
• Generalization and Stability of Interpolating Neural Networks with Minimal Width [37.908159361149835]
  We investigate the generalization and optimization of shallow neural-networks trained by gradient in the interpolating regime. We prove the training loss number minimizations $m=Omega(log4 (n))$ neurons and neurons $Tapprox n$. With $m=Omega(log4 (n))$ neurons and $Tapprox n$, we bound the test loss training by $tildeO (1/)$.
  arXiv  Detail & Related papers  (2023-02-18T05:06:15Z)
• Koopman-based generalization bound: New aspect for full-rank weights [45.64416573341108]
  We propose a new bound for generalization of neural networks using Koopman operators. Our result sheds new light on understanding generalization of neural networks with full-rank weight matrices.
  arXiv  Detail & Related papers  (2023-02-12T00:39:25Z)
• SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network [8.79431718760617]
  Training with mini-batch SGD and weight decay induces a bias toward rank minimization in weight matrices. We show that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. We empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.
  arXiv  Detail & Related papers  (2022-06-12T17:06:35Z)
• Towards an Understanding of Benign Overfitting in Neural Networks [104.2956323934544]
  Modern machine learning models often employ a huge number of parameters and are typically optimized to have zero training loss. We examine how these benign overfitting phenomena occur in a two-layer neural network setting. We show that it is possible for the two-layer ReLU network interpolator to achieve a near minimax-optimal learning rate.
  arXiv  Detail & Related papers  (2021-06-06T19:08:53Z)
• A case where a spindly two-layer linear network whips any neural network with a fully connected input layer [24.132345589750592]
  We show that a sparse input layer is needed to sample efficiently learn sparse targets with gradient descent. Surprisingly the same type of problem can be solved drastically more efficient by a simple 2-layer linear neural network.
  arXiv  Detail & Related papers  (2020-10-16T20:49:58Z)
• Towards Understanding Hierarchical Learning: Benefits of Neural Representations [160.33479656108926]
  In this work, we demonstrate that intermediate neural representations add more flexibility to neural networks. We show that neural representation can achieve improved sample complexities compared with the raw input. Our results characterize when neural representations are beneficial, and may provide a new perspective on why depth is important in deep learning.
  arXiv  Detail & Related papers  (2020-06-24T02:44:54Z)
• Revealing the Structure of Deep Neural Networks via Convex Duality [70.15611146583068]
  We study regularized deep neural networks (DNNs) and introduce a convex analytic framework to characterize the structure of hidden layers. We show that a set of optimal hidden layer weights for a norm regularized training problem can be explicitly found as the extreme points of a convex set. We apply the same characterization to deep ReLU networks with whitened data and prove the same weight alignment holds.
  arXiv  Detail & Related papers  (2020-02-22T21:13:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.