Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks

• URL: http://arxiv.org/abs/2604.09412v1
• Date: Fri, 10 Apr 2026 15:26:00 GMT
• Title: Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks
• Abstract summary: We study the population loss landscape of two-layer ReLU networks of the form $sum_k=1K mathrmReLU(w_ktop x)$ in a realisable teacher-student setting.<n>We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape.
• Score: 20.025943027616048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study the population loss landscape of two-layer ReLU networks of the form $\sum_{k=1}^K \mathrm{ReLU}(w_k^\top x)$ in a realisable teacher-student setting with Gaussian covariates. We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape. We further establish a direct link with one-pass SGD: local minima correspond to attractive fixed points of the dynamics in summary statistics space. This perspective reveals a hierarchical structure of minima: they are typically isolated in the well-specified regime, but become connected by flat directions as network width increases. In this overparameterised regime, global minima become increasingly accessible, attracting the dynamics and reducing convergence to spurious solutions. Overall, our results reveal intrinsic limitations of common simplifying assumptions, which may miss essential features of the loss landscape even in minimal neural network models.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• Benignity of loss landscape with weight decay requires both large overparametrization and initialization [9.680891499006469]
  We show that the landscape becomes benign -- free of spurious local minima -- under large overparametrization.<n>More precisely in this regime, almost all constant activation regions contain a global minimum and no spurious local minima.
  arXiv  Detail & Related papers  (2025-05-28T16:53:48Z)
• Simplicity Bias via Global Convergence of Sharpness Minimization [43.658859631741024]
  We show that label noise SGD always minimizes the sharpness on the manifold of models with zero loss for two-layer networks. We also find a novel property of the trace of Hessian of the loss at approximate stationary points on the manifold of zero loss.
  arXiv  Detail & Related papers  (2024-10-21T18:10:37Z)
• Generalization of Scaled Deep ResNets in the Mean-Field Regime [55.77054255101667]
  We investigate emphscaled ResNet in the limit of infinitely deep and wide neural networks. Our results offer new insights into the generalization ability of deep ResNet beyond the lazy training regime.
  arXiv  Detail & Related papers  (2024-03-14T21:48:00Z)
• Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features [33.703796571991745]
  Recent findings reveal that overparameterized deep neural networks, trained beyond zero training, exhibit a distinctive structural pattern at the final layer. These results indicate that the final-layer outputs in such networks display minimal within-class variations.
  arXiv  Detail & Related papers  (2024-02-29T06:02:45Z)
• Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization [4.418361486640713]
  We investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima.<n>We show how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows.
  arXiv  Detail & Related papers  (2023-09-01T14:53:51Z)
• Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape [8.425040193238777]
  We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. For one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima.
  arXiv  Detail & Related papers  (2023-05-31T02:49:13Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks [77.34726150561087]
  We show how to explore high-dimensional landscape characteristics of neural networks. We generalize observations on small neural networks to more complex systems. An interactive dashboard opens up a number of possible application networks.
  arXiv  Detail & Related papers  (2022-04-09T16:41:53Z)
• Neighborhood Region Smoothing Regularization for Finding Flat Minima In Deep Neural Networks [16.4654807047138]
  We propose an effective regularization technique, called Neighborhood Region Smoothing (NRS) NRS tries to regularize the neighborhood region in weight space to yield approximate outputs. We empirically show that the minima found by NRS would have relatively smaller Hessian eigenvalues compared to the conventional method.
  arXiv  Detail & Related papers  (2022-01-16T15:11:00Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.