Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization

• URL: http://arxiv.org/abs/2309.00508v4
• Date: Thu, 10 Apr 2025 13:56:57 GMT
• Title: Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization
• Abstract summary: We investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima.<n>We show how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows.
• Score: 4.418361486640713
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Under mild assumptions, we investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. Utilizing novel techniques, we demonstrate: (i) how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows; and (ii) the local convergence properties and rate of gradient flow dynamics. Our results indicate that two-layer neural networks can be locally recovered in the regime of overparameterization.

Related papers

• Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks [20.025943027616048]
  We study the population loss landscape of two-layer ReLU networks of the form $sum_k=1K mathrmReLU(w_ktop x)$ in a realisable teacher-student setting.<n>We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape.
  arXiv  Detail & Related papers  (2026-04-10T15:26:00Z)
• Low-Loss Space in Neural Networks is Continuous and Fully Connected [0.8212195887472242]
  We show that it is possible to connect two different minima with a path consisting of intermediate points that also have low loss.<n>Our work also provides new visualization methods and opportunities to improve model generalization.
  arXiv  Detail & Related papers  (2025-05-05T12:16:55Z)
• Generalization Error of Graph Neural Networks in the Mean-field Regime [10.35214360391282]
  We explore two widely utilized types of graph neural networks: graph convolutional neural networks and message passing graph neural networks. Our novel approach involves deriving upper bounds within the mean-field regime for evaluating the generalization error of these graph neural networks.
  arXiv  Detail & Related papers  (2024-02-10T19:12:31Z)
• Over-parameterised Shallow Neural Networks with Asymmetrical Node Scaling: Global Convergence Guarantees and Feature Learning [23.47570704524471]
  We consider optimisation of large and shallow neural networks via gradient flow, where the output of each hidden node is scaled by some positive parameter. We prove that, for large neural networks, with high probability, gradient flow converges to a global minimum AND can learn features, unlike in the NTK regime.
  arXiv  Detail & Related papers  (2023-02-02T10:40:06Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• FuNNscope: Visual microscope for interactively exploring the loss landscape of fully connected neural networks [77.34726150561087]
  We show how to explore high-dimensional landscape characteristics of neural networks. We generalize observations on small neural networks to more complex systems. An interactive dashboard opens up a number of possible application networks.
  arXiv  Detail & Related papers  (2022-04-09T16:41:53Z)
• Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometry [3.712728573432119]
  We develop a standardized parameterization in which all symmetries are removed, resulting in a toroidal topology. We derive a meaningful notion of the flatness of minimizers and of the geodesic paths connecting them. We also find that minimizers found by variants of gradient descent can be connected by zero-error paths with a single bend.
  arXiv  Detail & Related papers  (2022-02-07T09:57:54Z)
• An Unconstrained Layer-Peeled Perspective on Neural Collapse [20.75423143311858]
  We introduce a surrogate model called the unconstrained layer-peeled model (ULPM) We prove that gradient flow on this model converges to critical points of a minimum-norm separation problem exhibiting neural collapse in its global minimizer. We show that our results also hold during the training of neural networks in real-world tasks when explicit regularization or weight decay is not used.
  arXiv  Detail & Related papers  (2021-10-06T14:18:47Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• The Hidden Convex Optimization Landscape of Two-Layer ReLU Neural Networks: an Exact Characterization of the Optimal Solutions [51.60996023961886]
  We prove that finding all globally optimal two-layer ReLU neural networks can be performed by solving a convex optimization program with cone constraints. Our analysis is novel, characterizes all optimal solutions, and does not leverage duality-based analysis which was recently used to lift neural network training into convex spaces.
  arXiv  Detail & Related papers  (2020-06-10T15:38:30Z)
• A Mean-field Analysis of Deep ResNet and Beyond: Towards Provable Optimization Via Overparameterization From Depth [19.866928507243617]
  Training deep neural networks with gradient descent (SGD) can often achieve zero training loss on real-world landscapes. We propose a new limit of infinity deep residual networks, which enjoys a good training in the sense that everyr is global.
  arXiv  Detail & Related papers  (2020-03-11T20:14:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.