Estimate of the Neural Network Dimension using Algebraic Topology and Lie Theory

• URL: http://arxiv.org/abs/2004.02881v12
• Date: Mon, 16 Nov 2020 09:14:42 GMT
• Title: Estimate of the Neural Network Dimension using Algebraic Topology and Lie Theory
• Authors: Luciano Melodia, Richard Lenz
• Abstract summary: In this paper we present an approach to determine the smallest possible number of neurons in a layer of a neural network. We specify the required dimensions precisely, assuming that there is a smooth manifold on or near which the data are located. We derive the theory and validate it experimentally on toy data sets.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we present an approach to determine the smallest possible number of neurons in a layer of a neural network in such a way that the topology of the input space can be learned sufficiently well. We introduce a general procedure based on persistent homology to investigate topological invariants of the manifold on which we suspect the data set. We specify the required dimensions precisely, assuming that there is a smooth manifold on or near which the data are located. Furthermore, we require that this space is connected and has a commutative group structure in the mathematical sense. These assumptions allow us to derive a decomposition of the underlying space whose topology is well known. We use the representatives of the $k$-dimensional homology groups from the persistence landscape to determine an integer dimension for this decomposition. This number is the dimension of the embedding that is capable of capturing the topology of the data manifold. We derive the theory and validate it experimentally on toy data sets.

Related papers

• A Theoretical Study of Neural Network Expressive Power via Manifold Topology [9.054396245059555]
  A prevalent assumption regarding real-world data is that it lies on or close to a low-dimensional manifold. In this study, we investigate network expressive power in terms of the latent data manifold. We present a size upper bound of ReLU neural networks.
  arXiv  Detail & Related papers  (2024-10-21T22:10:24Z)
• A rank decomposition for the topological classification of neural representations [0.0]
  In this work, we leverage the fact that neural networks are equivalent to continuous piecewise-affine maps. We study the homology groups of the quotient of a manifold $mathcalM$ and a subset $A$, assuming some minimal properties on these spaces. We show that in randomly narrow networks, there will be regions in which the (co)homology groups of a data manifold can change.
  arXiv  Detail & Related papers  (2024-04-30T17:01:20Z)
• A Heat Diffusion Perspective on Geodesic Preserving Dimensionality Reduction [66.21060114843202]
  We propose a more general heat kernel based manifold embedding method that we call heat geodesic embeddings. Results show that our method outperforms existing state of the art in preserving ground truth manifold distances. We also showcase our method on single cell RNA-sequencing datasets with both continuum and cluster structure.
  arXiv  Detail & Related papers  (2023-05-30T13:58:50Z)
• Convolutional Neural Networks on Manifolds: From Graphs and Back [122.06927400759021]
  We propose a manifold neural network (MNN) composed of a bank of manifold convolutional filters and point-wise nonlinearities. To sum up, we focus on the manifold model as the limit of large graphs and construct MNNs, while we can still bring back graph neural networks by the discretization of MNNs.
  arXiv  Detail & Related papers  (2022-10-01T21:17:39Z)
• Intrinsic dimension estimation for discrete metrics [65.5438227932088]
  In this letter we introduce an algorithm to infer the intrinsic dimension (ID) of datasets embedded in discrete spaces. We demonstrate its accuracy on benchmark datasets, and we apply it to analyze a metagenomic dataset for species fingerprinting. This suggests that evolutive pressure acts on a low-dimensional manifold despite the high-dimensionality of sequences' space.
  arXiv  Detail & Related papers  (2022-07-20T06:38:36Z)
• Side-effects of Learning from Low Dimensional Data Embedded in an Euclidean Space [3.093890460224435]
  We study the potential regularization effects associated with the network's depth and noise in needs codimension of the data manifold. We also present additional side effects in training due to the presence of noise.
  arXiv  Detail & Related papers  (2022-03-01T16:55:51Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• The decomposition of the higher-order homology embedding constructed from the $k$-Laplacian [5.076419064097734]
  The null space of the $k$-th order Laplacian $mathbfmathcal L_k$ encodes the non-trivial topology of a manifold or a network. We propose an algorithm to factorize the homology embedding into subspaces corresponding to a manifold's simplest topological components.
  arXiv  Detail & Related papers  (2021-07-23T00:40:01Z)
• Intrinsic Dimension Estimation [92.87600241234344]
  We introduce a new estimator of the intrinsic dimension and provide finite sample, non-asymptotic guarantees. We then apply our techniques to get new sample complexity bounds for Generative Adversarial Networks (GANs) depending on the intrinsic dimension of the data.
  arXiv  Detail & Related papers  (2021-06-08T00:05:39Z)
• Manifold Density Estimation via Generalized Dequantization [9.090451761951101]
  Some kinds of data are not well-modeled by supposing that their underlying geometry is Euclidean. For instance, some kinds of data may be known to lie on the surface of a sphere. We propose a method, inspired by the literature on "dequantization," which we interpret through a coordinate transformation of an ambient Euclidean space.
  arXiv  Detail & Related papers  (2021-02-14T12:40:41Z)
• Disentangling by Subspace Diffusion [72.1895236605335]
  We show that fully unsupervised factorization of a data manifold is possible if the true metric of the manifold is known. Our work reduces the question of whether unsupervised metric learning is possible, providing a unifying insight into the geometric nature of representation learning.
  arXiv  Detail & Related papers  (2020-06-23T13:33:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.