Asymptotic Smoothing of the Lipschitz Loss Landscape in Overparameterized One-Hidden-Layer ReLU Networks

• URL: http://arxiv.org/abs/2602.17596v1
• Date: Thu, 19 Feb 2026 18:20:21 GMT
• Title: Asymptotic Smoothing of the Lipschitz Loss Landscape in Overparameterized One-Hidden-Layer ReLU Networks
• Abstract summary: We show that every pair of models at the same loss level can be connected by a continuous path within an arbitrarily small loss.<n>On a synthetic Moons dataset and on the Wisconsin Breast Cancer dataset, we find that wider networks exhibit smaller gaps.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the topology of the loss landscape of one-hidden-layer ReLU networks under overparameterization. On the theory side, we (i) prove that for convex $L$-Lipschitz losses with an $\ell_1$-regularized second layer, every pair of models at the same loss level can be connected by a continuous path within an arbitrarily small loss increase $ε$ (extending a known result for the quadratic loss); (ii) obtain an asymptotic upper bound on the energy gap $ε$ between local and global minima that vanishes as the width $m$ grows, implying that the landscape flattens and sublevel sets become connected in the limit. Empirically, on a synthetic Moons dataset and on the Wisconsin Breast Cancer dataset, we measure pairwise energy gaps via Dynamic String Sampling (DSS) and find that wider networks exhibit smaller gaps; in particular, a permutation test on the maximum gap yields $p_{perm}=0$, indicating a clear reduction in the barrier height.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• SILAGE: Memory-Efficient, Full-Gradient-Free Nonconvex Optimization for Nested Finite Sums [51.49970814177172]
  Empirical risk on datasets naturally exhibits similarities in $N=nm$ total samples.<n>We provide an analysis that avoids pessimistic convergence analysis.<n>Our results improve existing state-of-the-art regimes.
  arXiv  Detail & Related papers  (2026-06-14T14:11:07Z)
• Optimal Dimension-Free Sampling for Regularized Classification [56.72526267755301]
  We prove optimal sampling bounds achieving $(1pmvarepsilon)$-relative error for a broad class of Lipschitz continuous classification loss functions.<n>This includes important functions such as logistic and sigmoid loss, hinge loss, and ReLU loss, as prominent and popular representative examples.
  arXiv  Detail & Related papers  (2026-05-22T15:05:33Z)
• Minimax Rates for Hyperbolic Hierarchical Learning [3.3192479135000426]
  We prove an exponential separation in sample complexity between Euclidean and hyperbolic representations for learning on hierarchical data.<n>We also show a geometry-independent bottleneck: any rank-$k$ prediction space captures only $O(k)$ canonical hierarchical contrasts.
  arXiv  Detail & Related papers  (2026-01-27T20:50:24Z)
• Emergence and scaling laws in SGD learning of shallow neural networks [64.48316762675141]
  We study the complexity of online gradient descent (SGD) for learning a two-layer neural network with $P$ neurons on isotropic Gaussian data.<n>We provide a precise analysis of SGD dynamics for the training of a student two-layer network to minimize the mean squared error (MSE) objective.
  arXiv  Detail & Related papers  (2025-04-28T16:58:55Z)
• Convergence of Shallow ReLU Networks on Weakly Interacting Data [5.618969269882913]
  We analyse the convergence of one-hidden-layer ReLU networks trained by gradient flow on $n$ data points.<n>We show that a network with width of order $log(n)$ neurons suffices for global convergence with high probability.
  arXiv  Detail & Related papers  (2025-02-24T09:07:14Z)
• Criteria and Bias of Parameterized Linear Regression under Edge of Stability Regime [38.134523847923646]
  Edge of Stability (EoS) is usually known as the Edge of Stability (EoS) phenomenon.<n>We show that EoS occurs even when $l$ is quadratic under proper conditions.<n>We also shed some new light on the implicit bias of diagonal linear networks when a larger step-size is employed.
  arXiv  Detail & Related papers  (2024-12-11T02:07:37Z)
• Information-Theoretic Thresholds for Planted Dense Cycles [52.076657911275525]
  We study a random graph model for small-world networks which are ubiquitous in social and biological sciences. For both detection and recovery of the planted dense cycle, we characterize the information-theoretic thresholds in terms of $n$, $tau$, and an edge-wise signal-to-noise ratio $lambda$.
  arXiv  Detail & Related papers  (2024-02-01T03:39:01Z)
• Noisy Interpolation Learning with Shallow Univariate ReLU Networks [33.900009202637285]
  Mallinar et. al. 2022 noted that neural networks seem to often exhibit tempered overfitting'', wherein the population risk does not converge to the Bayes optimal error. We provide the first rigorous analysis of the overfitting behavior of regression with minimum weights.
  arXiv  Detail & Related papers  (2023-07-28T08:41:12Z)
• The Implicit Bias of Minima Stability in Multivariate Shallow ReLU Networks [53.95175206863992]
  We study the type of solutions to which gradient descent converges when used to train a single hidden-layer multivariate ReLU network with the quadratic loss. We prove that although shallow ReLU networks are universal approximators, stable shallow networks are not.
  arXiv  Detail & Related papers  (2023-06-30T09:17:39Z)
• Detection-Recovery Gap for Planted Dense Cycles [72.4451045270967]
  We consider a model where a dense cycle with expected bandwidth $n tau$ and edge density $p$ is planted in an ErdHos-R'enyi graph $G(n,q)$. We characterize the computational thresholds for the associated detection and recovery problems for the class of low-degree algorithms.
  arXiv  Detail & Related papers  (2023-02-13T22:51:07Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Analytic Study of Families of Spurious Minima in Two-Layer ReLU Neural Networks [15.711517003382484]
  We show that the Hessian spectrum concentrates near positives, with the exception of $Theta(d)$ eigenvalues which growly with$d$. This makes possible the creation and the annihilation of minima using powerful tools from bifurcation theory.
  arXiv  Detail & Related papers  (2021-07-21T22:05:48Z)
• Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariances [9.390008801320024]
  We show that adding one extra neuron to each is sufficient to connect all previously discrete minima into a single manifold. We show that the number of symmetry-induced critical subspaces dominates the number of affine subspaces forming the global minima manifold.
  arXiv  Detail & Related papers  (2021-05-25T21:19:07Z)
• Agnostic Learning of Halfspaces with Gradient Descent via Soft Margins [92.7662890047311]
  We show that gradient descent finds halfspaces with classification error $tilde O(mathsfOPT1/2) + varepsilon$ in $mathrmpoly(d,1/varepsilon)$ time and sample complexity.
  arXiv  Detail & Related papers  (2020-10-01T16:48:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.