Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective

• URL: http://arxiv.org/abs/2001.06931v2
• Date: Wed, 18 Mar 2020 01:00:37 GMT
• Title: Any Target Function Exists in a Neighborhood of Any Sufficiently Wide Random Network: A Geometrical Perspective
• Authors: Shun-ichi Amari
• Abstract summary: It is known that any target function is realized in a sufficiently small neighborhood of any randomly connected deep network. We show that high-dimensional geometry plays a magical role.
• Score: 4.42494528420519
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It is known that any target function is realized in a sufficiently small neighborhood of any randomly connected deep network, provided the width (the number of neurons in a layer) is sufficiently large. There are sophisticated theories and discussions concerning this striking fact, but rigorous theories are very complicated. We give an elementary geometrical proof by using a simple model for the purpose of elucidating its structure. We show that high-dimensional geometry plays a magical role: When we project a high-dimensional sphere of radius 1 to a low-dimensional subspace, the uniform distribution over the sphere reduces to a Gaussian distribution of negligibly small covariances.

Related papers

• Deep Generative Models: Complexity, Dimensionality, and Approximation [18.530164193967945]
  We show that generative networks can generate high-dimensional data from lower-dimensional inputs. Our findings challenge the belief on the relationship between input dimensionality and the ability of generative networks to model data distributions. This novel insight not only corroborates the practical effectiveness of generative networks in handling complex data structures, but also underscores a critical trade-off between approximation error, dimensionality, and model complexity.
  arXiv  Detail & Related papers  (2025-04-01T14:07:02Z)
• A Theoretical Study of Neural Network Expressive Power via Manifold Topology [9.054396245059555]
  A prevalent assumption regarding real-world data is that it lies on or close to a low-dimensional manifold. In this study, we investigate network expressive power in terms of the latent data manifold. We present a size upper bound of ReLU neural networks.
  arXiv  Detail & Related papers  (2024-10-21T22:10:24Z)
• Decoder ensembling for learned latent geometries [15.484595752241122]
  We show how to easily compute geodesics on the associated expected manifold. We find this simple and reliable, thereby coming one step closer to easy-to-use latent geometries.
  arXiv  Detail & Related papers  (2024-08-14T12:35:41Z)
• Scaling Riemannian Diffusion Models [68.52820280448991]
  We show that our method enables us to scale to high dimensional tasks on nontrivial manifold. We model QCD densities on $SU(n)$ lattices and contrastively learned embeddings on high dimensional hyperspheres.
  arXiv  Detail & Related papers  (2023-10-30T21:27:53Z)
• On Deep Generative Models for Approximation and Estimation of Distributions on Manifolds [38.311376714689]
  Generative networks can generate high-dimensional complex data from a low-dimensional easy-to-sample distribution. We take such low-dimensional data structures into consideration by assuming that data distributions are supported on a low-dimensional manifold. We show that the Wasserstein-1 loss converges to zero at a fast rate depending on the intrinsic dimension instead of the ambient data dimension.
  arXiv  Detail & Related papers  (2023-02-25T22:34:19Z)
• Implications of sparsity and high triangle density for graph representation learning [67.98498239263549]
  Recent work has shown that sparse graphs containing many triangles cannot be reproduced using a finite-dimensional representation of the nodes. Here, we show that such graphs can be reproduced using an infinite-dimensional inner product model, where the node representations lie on a low-dimensional manifold.
  arXiv  Detail & Related papers  (2022-10-27T09:15:15Z)
• A singular Riemannian geometry approach to Deep Neural Networks I. Theoretical foundations [77.86290991564829]
  Deep Neural Networks are widely used for solving complex problems in several scientific areas, such as speech recognition, machine translation, image analysis. We study a particular sequence of maps between manifold, with the last manifold of the sequence equipped with a Riemannian metric. We investigate the theoretical properties of the maps of such sequence, eventually we focus on the case of maps between implementing neural networks of practical interest.
  arXiv  Detail & Related papers  (2021-12-17T11:43:30Z)
• Dist2Cycle: A Simplicial Neural Network for Homology Localization [66.15805004725809]
  Simplicial complexes can be viewed as high dimensional generalizations of graphs that explicitly encode multi-way ordered relations. We propose a graph convolutional model for learning functions parametrized by the $k$-homological features of simplicial complexes.
  arXiv  Detail & Related papers  (2021-10-28T14:59:41Z)
• Nested-sphere description of the N-level Chern number and the generalized Bloch hypersphere [0.0]
  The geometric interpretation of (pseudo)spin 1/2 systems on the Bloch sphere has been appreciated across different areas ranging from condensed matter to quantum information and high energy physics. We here employ a coherence vector description to theoretically characterize a general N-level system on the higher dimensional generalized Bloch (hyper)sphere. We demonstrate that for the N-level case, there is an exterior two-sphere that provides a useful characterization of the system, notably by playing a primary role in determining the Chern number.
  arXiv  Detail & Related papers  (2021-10-13T18:00:01Z)
• Intrinsic Dimension Estimation [92.87600241234344]
  We introduce a new estimator of the intrinsic dimension and provide finite sample, non-asymptotic guarantees. We then apply our techniques to get new sample complexity bounds for Generative Adversarial Networks (GANs) depending on the intrinsic dimension of the data.
  arXiv  Detail & Related papers  (2021-06-08T00:05:39Z)
• From Geometry to Topology: Inverse Theorems for Distributed Persistence [0.0]
  We show that the correct invariant is not the persistence diagram of X, but rather the collection of persistence diagrams of many small subsets. This invariant, which we call "distributed persistence," is trivially parallelizable, more stable to outliers. Results are complemented by two synthetic experiments demonstrating the use of distributed persistence in practice.
  arXiv  Detail & Related papers  (2021-01-28T21:36:45Z)
• A simple geometric proof for the benefit of depth in ReLU networks [57.815699322370826]
  We present a simple proof for the benefit of depth in multi-layer feedforward network with rectified activation ("depth separation") We present a concrete neural network with linear depth (in $m$) and small constant width ($leq 4$) that classifies the problem with zero error.
  arXiv  Detail & Related papers  (2021-01-18T15:40:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.