Sufficient Conditions for Stability of Minimum-Norm Interpolating Deep ReLU Networks

• URL: http://arxiv.org/abs/2602.13910v1
• Date: Sat, 14 Feb 2026 22:20:44 GMT
• Title: Sufficient Conditions for Stability of Minimum-Norm Interpolating Deep ReLU Networks
• Abstract summary: We study the algorithmic stability of deep homogeneous neural networks that achieve zero training error using parameters with the smallest $L$ norm.<n>We find that 1) such networks are stable when they contain a stable sub-network, followed by a layer with a low-rank weight matrix, and 2) such networks are not guaranteed to be stable even when they contain a stable sub-network, if the following layer is not low-rank.
• Score: 18.714326830182735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Algorithmic stability is a classical framework for analyzing the generalization error of learning algorithms. It predicts that an algorithm has small generalization error if it is insensitive to small perturbations in the training set such as the removal or replacement of a training point. While stability has been demonstrated for numerous well-known algorithms, this framework has had limited success in analyses of deep neural networks. In this paper we study the algorithmic stability of deep ReLU homogeneous neural networks that achieve zero training error using parameters with the smallest $L_2$ norm, also known as the minimum-norm interpolation, a phenomenon that can be observed in overparameterized models trained by gradient-based algorithms. We investigate sufficient conditions for such networks to be stable. We find that 1) such networks are stable when they contain a (possibly small) stable sub-network, followed by a layer with a low-rank weight matrix, and 2) such networks are not guaranteed to be stable even when they contain a stable sub-network, if the following layer is not low-rank. The low-rank assumption is inspired by recent empirical and theoretical results which demonstrate that training deep neural networks is biased towards low-rank weight matrices, for minimum-norm interpolation and weight-decay regularization.

Related papers

• On the Stability of the Jacobian Matrix in Deep Neural Networks [9.617753464544606]
  We establish a general stability theorem for deep neural networks that accommodates sparsity and weakly correlated weights.<n>Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models.
  arXiv  Detail & Related papers  (2025-06-10T13:04:42Z)
• Low-rank bias, weight decay, and model merging in neural networks [12.352761060862072]
  We show several properties of such deep neural networks, induced by $L2$ regularization.<n>In addition, we investigate a multitask learning phenomenon enabled by $L2$ regularization and low-rank bias.
  arXiv  Detail & Related papers  (2025-02-24T17:17:00Z)
• Stabilizing RNN Gradients through Pre-training [3.335932527835653]
  Theory of learning proposes to prevent the gradient from exponential growth with depth or time, to stabilize and improve training. We extend known stability theories to encompass a broader family of deep recurrent networks, requiring minimal assumptions on data and parameter distribution. We propose a new approach to mitigate this issue, that consists on giving a weight of a half to the time and depth contributions to the gradient.
  arXiv  Detail & Related papers  (2023-08-23T11:48:35Z)
• Computational Complexity of Learning Neural Networks: Smoothness and Degeneracy [52.40331776572531]
  We show that learning depth-$3$ ReLU networks under the Gaussian input distribution is hard even in the smoothed-analysis framework. Our results are under a well-studied assumption on the existence of local pseudorandom generators.
  arXiv  Detail & Related papers  (2023-02-15T02:00:26Z)
• Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach [64.23331120621118]
  This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks. We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network. We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
  arXiv  Detail & Related papers  (2022-08-08T03:13:24Z)
• On the Intrinsic Structures of Spiking Neural Networks [66.57589494713515]
  Recent years have emerged a surge of interest in SNNs owing to their remarkable potential to handle time-dependent and event-driven data. There has been a dearth of comprehensive studies examining the impact of intrinsic structures within spiking computations. This work delves deep into the intrinsic structures of SNNs, by elucidating their influence on the expressivity of SNNs.
  arXiv  Detail & Related papers  (2022-06-21T09:42:30Z)
• A Kernel-Expanded Stochastic Neural Network [10.837308632004644]
  Deep neural network often gets trapped into a local minimum in training. New kernel-expanded neural network (K-StoNet) model reformulates the network as a latent variable model. Model can be easily trained using the imputationregularized optimization (IRO) algorithm.
  arXiv  Detail & Related papers  (2022-01-14T06:42:42Z)
• Training Certifiably Robust Neural Networks with Efficient Local Lipschitz Bounds [99.23098204458336]
  Certified robustness is a desirable property for deep neural networks in safety-critical applications. We show that our method consistently outperforms state-of-the-art methods on MNIST and TinyNet datasets.
  arXiv  Detail & Related papers  (2021-11-02T06:44:10Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)
• LocalDrop: A Hybrid Regularization for Deep Neural Networks [98.30782118441158]
  We propose a new approach for the regularization of neural networks by the local Rademacher complexity called LocalDrop. A new regularization function for both fully-connected networks (FCNs) and convolutional neural networks (CNNs) has been developed based on the proposed upper bound of the local Rademacher complexity.
  arXiv  Detail & Related papers  (2021-03-01T03:10:11Z)
• On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory [77.18089185140767]
  Lipschitz regularity is established as a key property of modern deep learning. computing the exact value of the Lipschitz constant of a neural network is known to be NP-hard. We introduce a new upper bound for convolutional layers that is both tight and easy to compute.
  arXiv  Detail & Related papers  (2020-06-15T13:23:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.