Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape

• URL: http://arxiv.org/abs/2305.19510v3
• Date: Thu, 8 Feb 2024 15:43:22 GMT
• Title: Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape
• Abstract summary: We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. For one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima.
• Score: 8.425040193238777
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the loss landscape of both shallow and deep, mildly overparameterized ReLU neural networks on a generic finite input dataset for the squared error loss. We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. Furthermore, for one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima. We experimentally confirm these results by finding a phase transition from most regions having full rank Jacobian to many regions having deficient rank depending on the amount of overparameterization.

Related papers

• Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks [20.025943027616048]
  We study the population loss landscape of two-layer ReLU networks of the form $sum_k=1K mathrmReLU(w_ktop x)$ in a realisable teacher-student setting.<n>We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape.
  arXiv  Detail & Related papers  (2026-04-10T15:26:00Z)
• Diffusion-based Data Augmentation for Object Counting Problems [62.63346162144445]
  We develop a pipeline that utilizes a diffusion model to generate extensive training data. We are the first to generate images conditioned on a location dot map with a diffusion model. Our proposed counting loss for the diffusion model effectively minimizes the discrepancies between the location dot map and the crowd images generated.
  arXiv  Detail & Related papers  (2024-01-25T07:28:22Z)
• Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization [4.418361486640713]
  We investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. We show how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows.
  arXiv  Detail & Related papers  (2023-09-01T14:53:51Z)
• Semi-signed neural fitting for surface reconstruction from unoriented point clouds [53.379712818791894]
  We propose SSN-Fitting to reconstruct a better signed distance field. SSN-Fitting consists of a semi-signed supervision and a loss-based region sampling strategy. We conduct experiments to demonstrate that SSN-Fitting achieves state-of-the-art performance under different settings.
  arXiv  Detail & Related papers  (2022-06-14T09:40:17Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• The loss landscape of deep linear neural networks: a second-order analysis [9.85879905918703]
  We study the optimization landscape of deep linear neural networks with the square loss. We characterize, among all critical points, which are global minimizers, strict saddle points, and non-strict saddle points.
  arXiv  Detail & Related papers  (2021-07-28T11:33:18Z)
• Topological obstructions in neural networks learning [67.8848058842671]
  We study global properties of the loss gradient function flow. We use topological data analysis of the loss function and its Morse complex to relate local behavior along gradient trajectories with global properties of the loss surface.
  arXiv  Detail & Related papers  (2020-12-31T18:53:25Z)
• Fully reversible neural networks for large-scale surface and sub-surface characterization via remote sensing [4.383011485317949]
  Large spatial/frequency scale of hyperspectral and airborne magnetic and gravitational data causes memory issues when using convolutional neural networks for (sub-) surface characterization. We show examples from land-use change detection from hyperspectral time-lapse data, and regional aquifer mapping from airborne geophysical and geological data.
  arXiv  Detail & Related papers  (2020-03-16T23:54:22Z)
• Beyond Dropout: Feature Map Distortion to Regularize Deep Neural Networks [107.77595511218429]
  In this paper, we investigate the empirical Rademacher complexity related to intermediate layers of deep neural networks. We propose a feature distortion method (Disout) for addressing the aforementioned problem. The superiority of the proposed feature map distortion for producing deep neural network with higher testing performance is analyzed and demonstrated.
  arXiv  Detail & Related papers  (2020-02-23T13:59:13Z)
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory [4.976129960952446]
  For one-hidden-layer ReLU networks, we prove that all differentiable local minima are global inside differentiable regions. We show that these local minima can be isolated points or continuous hyperplanes, depending on an interplay between data, activation pattern of hidden neurons and network size.
  arXiv  Detail & Related papers  (2020-02-12T02:04:55Z)
• Barcodes as Summary of Loss Function Topology [65.3479573549873]
  We show that increase of the neural network's depth and width lowers the barcodes of local minima. This has some natural implications for the neural network's learning and for its generalization properties.
  arXiv  Detail & Related papers  (2019-11-29T19:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.