Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory

• URL: http://arxiv.org/abs/2002.04763v2
• Date: Wed, 17 Jun 2020 01:57:24 GMT
• Title: Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 1: Theory
• Abstract summary: For one-hidden-layer ReLU networks, we prove that all differentiable local minima are global inside differentiable regions. We show that these local minima can be isolated points or continuous hyperplanes, depending on an interplay between data, activation pattern of hidden neurons and network size.
• Score: 4.976129960952446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: For one-hidden-layer ReLU networks, we prove that all differentiable local minima are global inside differentiable regions. We give the locations and losses of differentiable local minima, and show that these local minima can be isolated points or continuous hyperplanes, depending on an interplay between data, activation pattern of hidden neurons and network size. Furthermore, we give necessary and sufficient conditions for the existence of saddle points as well as non-differentiable local minima, and their locations if they exist.

Related papers

• Sharp description of local minima in the loss landscape of high-dimensional two-layer ReLU neural networks [20.025943027616048]
  We study the population loss landscape of two-layer ReLU networks of the form $sum_k=1K mathrmReLU(w_ktop x)$ in a realisable teacher-student setting.<n>We show that local minima admit an exact low-dimensional representation in terms of summary statistics, yielding a sharp and interpretable characterisation of the landscape.
  arXiv  Detail & Related papers  (2026-04-10T15:26:00Z)
• Loss Landscape of Shallow ReLU-like Neural Networks: Stationary Points, Saddle Escaping, and Network Embedding [1.4513150969598634]
  We investigate the loss landscape of one-hidden-layer neural networks with ReLU-like activation functions trained with the empirical squared loss. As the activation function is non-differentiable, it is so far unclear how to completely characterize the stationary points. We show that, if a stationary point does not contain "escape neurons", which are defined with first-order conditions, then it must be a local minimum.
  arXiv  Detail & Related papers  (2024-02-08T12:30:29Z)
• Geometry and Local Recovery of Global Minima of Two-layer Neural Networks at Overparameterization [4.418361486640713]
  We investigate the geometry of the loss landscape for two-layer neural networks in the vicinity of global minima. We show how global minima with zero generalization error become geometrically separated from other global minima as the sample size grows.
  arXiv  Detail & Related papers  (2023-09-01T14:53:51Z)
• Mildly Overparameterized ReLU Networks Have a Favorable Loss Landscape [8.425040193238777]
  We show both by count and volume that most activation patterns correspond to parameter regions with no bad local minima. For one-dimensional input data, we show most activation regions realizable by the network contain a high dimensional set of global minima and no bad local minima.
  arXiv  Detail & Related papers  (2023-05-31T02:49:13Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• On the Omnipresence of Spurious Local Minima in Certain Neural Network Training Problems [0.0]
  We study the loss landscape of training problems for deep artificial neural networks with a one-dimensional real output. It is shown that such problems possess a continuum of spurious (i.e., not globally optimal) local minima for all target functions that are not affine.
  arXiv  Detail & Related papers  (2022-02-23T14:41:54Z)
• Understanding Global Loss Landscape of One-hidden-layer ReLU Networks, Part 2: Experiments and Analysis [4.976129960952446]
  We first analyze how big the probability of existing local minima is for 1D Gaussian data and how it varies in the whole weight space. We then design and implement a linear programming based approach to judge the existence of genuine local minima. We find that there are no bad differentiable local minima almost everywhere in weight space once some hidden neurons are activated by samples.
  arXiv  Detail & Related papers  (2020-06-15T10:50:47Z)
• LSM: Learning Subspace Minimization for Low-level Vision [78.27774638569218]
  We replace the regularization term with a learnable subspace constraint, and preserve the data term to exploit domain knowledge. This learning subspace minimization (LSM) framework unifies the network structures and the parameters for many low-level vision tasks. We demonstrate our LSM framework on four low-level tasks including interactive image segmentation, video segmentation, stereo matching, and optical flow, and validate the network on various datasets.
  arXiv  Detail & Related papers  (2020-04-20T10:49:38Z)
• Piecewise linear activations substantially shape the loss surfaces of neural networks [95.73230376153872]
  This paper presents how piecewise linear activation functions substantially shape the loss surfaces of neural networks. We first prove that it the loss surfaces of many neural networks have infinite spurious local minima which are defined as the local minima with higher empirical risks than the global minima. For one-hidden-layer networks, we prove that all local minima in a cell constitute an equivalence class; they are concentrated in a valley; and they are all global minima in the cell.
  arXiv  Detail & Related papers  (2020-03-27T04:59:34Z)
• LRC-Net: Learning Discriminative Features on Point Clouds by Encoding Local Region Contexts [65.79931333193016]
  We present a novel Local-Region-Context Network (LRC-Net) to learn discriminative features on point clouds. LRC-Net encodes fine-grained contexts inside and among local regions simultaneously. Results show LRC-Net is competitive with state-of-the-art methods in shape classification and shape segmentation applications.
  arXiv  Detail & Related papers  (2020-03-18T14:34:08Z)
• The Devil is in the Channels: Mutual-Channel Loss for Fine-Grained Image Classification [67.79883226015824]
  Key for solving fine-grained image categorization is finding discriminate and local regions that correspond to subtle visual traits. In this paper, we show it is possible to cultivate subtle details without the need for overly complicated network designs or training mechanisms. The proposed loss function, termed as mutual-channel loss (MC-Loss), consists of two channel-specific components.
  arXiv  Detail & Related papers  (2020-02-11T09:12:45Z)
• Dense Residual Network: Enhancing Global Dense Feature Flow for Character Recognition [75.4027660840568]
  This paper explores how to enhance the local and global dense feature flow by exploiting hierarchical features fully from all the convolution layers. Technically, we propose an efficient and effective CNN framework, i.e., Fast Dense Residual Network (FDRN) for text recognition.
  arXiv  Detail & Related papers  (2020-01-23T06:55:08Z)
• Barcodes as Summary of Loss Function Topology [65.3479573549873]
  We show that increase of the neural network's depth and width lowers the barcodes of local minima. This has some natural implications for the neural network's learning and for its generalization properties.
  arXiv  Detail & Related papers  (2019-11-29T19:22:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.