Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 2: Experiments and Analysis

• URL: http://arxiv.org/abs/2006.09192v1
• Date: Mon, 15 Jun 2020 10:50:47 GMT
• Title: Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 2: Experiments and Analysis
• Abstract summary: We first analyze how big the probability of existing local minima is for 1D Gaussian data and how it varies in the whole weight space. We then design and implement a linear programming based approach to judge the existence of genuine local minima. We find that there are no bad differentiable local minima almost everywhere in weight space once some hidden neurons are activated by samples.
• Score: 4.976129960952446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The existence of local minima for one-hidden-layer ReLU networks has been investigated theoretically in [8]. Based on the theory, in this paper, we first analyze how big the probability of existing local minima is for 1D Gaussian data and how it varies in the whole weight space. We show that this probability is very low in most regions. We then design and implement a linear programming based approach to judge the existence of genuine local minima, and use it to predict whether bad local minima exist for the MNIST and CIFAR-10 datasets, and find that there are no bad differentiable local minima almost everywhere in weight space once some hidden neurons are activated by samples. These theoretical predictions are verified experimentally by showing that gradient descent is not trapped in the cells from which it starts. We also perform experiments to explore the count and size of differentiable cells in the weight space.

Related papers

• Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations [53.348574336527854]
  We investigate the neural network map as a function on the space of weights and biases.<n>We prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures.
  arXiv  Detail & Related papers  (2025-10-28T14:34:33Z)
• Local Poisson Deconvolution for Discrete Signals [2.1374208474242815]
  We analyze the problem of recovering an atomic signal from a binned Poisson convolution model.<n>Our main results quantify the local minimax risk of estimating $mu$ for a broad class of smooth convolution kernels.<n>These results paint an optimistic perspective on the Poisson deconvolution problem.
  arXiv  Detail & Related papers  (2025-08-01T17:59:57Z)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon [22.29950158991071]
  We study the implicit bias of flatness / low (loss) curvature and its effects on generalization in ReLU networks.<n>We show that while flatness does imply generalization, the resulting rates of convergence necessarily deteriorate exponentially as the input dimension grows.
  arXiv  Detail & Related papers  (2025-06-25T19:10:03Z)
• Just How Flexible are Neural Networks in Practice? [89.80474583606242]
  It is widely believed that a neural network can fit a training set containing at least as many samples as it has parameters. In practice, however, we only find solutions via our training procedure, including the gradient and regularizers, limiting flexibility.
  arXiv  Detail & Related papers  (2024-06-17T12:24:45Z)
• When Expressivity Meets Trainability: Fewer than $n$ Neurons Can Work [59.29606307518154]
  We show that as long as the width $m geq 2n/d$ (where $d$ is the input dimension), its expressivity is strong, i.e., there exists at least one global minimizer with zero training loss. We also consider a constrained optimization formulation where the feasible region is the nice local region, and prove that every KKT point is a nearly global minimizer.
  arXiv  Detail & Related papers  (2022-10-21T14:41:26Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Neighborhood Region Smoothing Regularization for Finding Flat Minima In Deep Neural Networks [16.4654807047138]
  We propose an effective regularization technique, called Neighborhood Region Smoothing (NRS) NRS tries to regularize the neighborhood region in weight space to yield approximate outputs. We empirically show that the minima found by NRS would have relatively smaller Hessian eigenvalues compared to the conventional method.
  arXiv  Detail & Related papers  (2022-01-16T15:11:00Z)
• LDC-Net: A Unified Framework for Localization, Detection and Counting in Dense Crowds [103.8635206945196]
  The rapid development in visual crowd analysis shows a trend to count people by positioning or even detecting, rather than simply summing a density map. Some recent work on crowd localization and detection has two limitations: 1) The typical detection methods can not handle the dense crowds and a large variation in scale; 2) The density map methods suffer from performance deficiency in position and box prediction, especially in high density or large-size crowds.
  arXiv  Detail & Related papers  (2021-10-10T07:55:44Z)
• Unveiling the structure of wide flat minima in neural networks [0.46664938579243564]
  Deep learning has revealed the application potential of networks across the sciences. The success of deep learning has revealed the application potential of networks across the sciences.
  arXiv  Detail & Related papers  (2021-07-02T16:04:57Z)
• Effective Version Space Reduction for Convolutional Neural Networks [61.84773892603885]
  In active learning, sampling bias could pose a serious inconsistency problem and hinder the algorithm from finding the optimal hypothesis. We examine active learning with convolutional neural networks through the principled lens of version space reduction.
  arXiv  Detail & Related papers  (2020-06-22T17:40:03Z)
• How Many Samples is a Good Initial Point Worth in Low-rank Matrix Recovery? [12.589519278962378]
  A non-low-rank matrix recovery problem contains no spurious local minima. We quantify the relationship between the quality of the initial guess and the corresponding reduction in data requirements.
  arXiv  Detail & Related papers  (2020-06-12T02:47:03Z)
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory [4.976129960952446]
  For one-hidden-layer ReLU networks, we prove that all differentiable local minima are global inside differentiable regions. We show that these local minima can be isolated points or continuous hyperplanes, depending on an interplay between data, activation pattern of hidden neurons and network size.
  arXiv  Detail & Related papers  (2020-02-12T02:04:55Z)
• A Diffusion Theory For Deep Learning Dynamics: Stochastic Gradient Descent Exponentially Favors Flat Minima [91.11332770406007]
  We show that Gradient Descent (SGD) favors flat minima exponentially more than sharp minima. We also reveal that either a small learning rate or large-batch training requires exponentially many iterations to escape from minima.
  arXiv  Detail & Related papers  (2020-02-10T02:04:49Z)
• Barcodes as Summary of Loss Function Topology [65.3479573549873]
  We show that increase of the neural network's depth and width lowers the barcodes of local minima. This has some natural implications for the neural network's learning and for its generalization properties.
  arXiv  Detail & Related papers  (2019-11-29T19:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.