Metric Flows with Neural Networks
- URL: http://arxiv.org/abs/2310.19870v1
- Date: Mon, 30 Oct 2023 18:00:01 GMT
- Title: Metric Flows with Neural Networks
- Authors: James Halverson and Fabian Ruehle
- Abstract summary: We develop a theory of flows induced by neural network gradient descent.
This is motivated in part by advances in approximating Calabi-Yau metrics with neural networks.
We apply these ideas to numerical Calabi-Yau metrics, including a discussion on the importance of feature learning.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We develop a theory of flows in the space of Riemannian metrics induced by
neural network gradient descent. This is motivated in part by recent advances
in approximating Calabi-Yau metrics with neural networks and is enabled by
recent advances in understanding flows in the space of neural networks. We
derive the corresponding metric flow equations, which are governed by a metric
neural tangent kernel, a complicated, non-local object that evolves in time.
However, many architectures admit an infinite-width limit in which the kernel
becomes fixed and the dynamics simplify. Additional assumptions can induce
locality in the flow, which allows for the realization of Perelman's
formulation of Ricci flow that was used to resolve the 3d Poincar\'e
conjecture. We apply these ideas to numerical Calabi-Yau metrics, including a
discussion on the importance of feature learning.
Related papers
- Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence.
We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers.
This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
arXiv Detail & Related papers (2023-07-20T13:34:11Z) - Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights.
We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
arXiv Detail & Related papers (2023-02-01T03:18:07Z) - Simple initialization and parametrization of sinusoidal networks via
their kernel bandwidth [92.25666446274188]
sinusoidal neural networks with activations have been proposed as an alternative to networks with traditional activation functions.
We first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis.
We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth.
arXiv Detail & Related papers (2022-11-26T07:41:48Z) - Proximal Mean Field Learning in Shallow Neural Networks [0.4972323953932129]
We propose a custom learning algorithm for shallow neural networks with single hidden layer having infinite width.
We realize mean field learning as a computational algorithm, rather than as an analytical tool.
Our algorithm performs gradient descent of the free energy associated with the risk functional.
arXiv Detail & Related papers (2022-10-25T10:06:26Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - Approximation Power of Deep Neural Networks: an explanatory mathematical
survey [0.0]
The goal of this survey is to present an explanatory review of the approximation properties of deep neural networks.
We aim at understanding how and why deep neural networks outperform other classical linear and nonlinear approximation methods.
arXiv Detail & Related papers (2022-07-19T18:47:44Z) - Dynamically Stable Poincar\'e Embeddings for Neural Manifolds [10.76554740227876]
We prove that if initial metrics have an $L2$-norm perturbation which deviates from the Hyperbolic metric on the Poincar'e ball, the scaled Ricci-DeTurck flow of such metrics smoothly and exponentially converges to the Hyperbolic metric.
Specifically, the role of the Ricci flow is to serve as naturally evolving to the stable Poincar'e ball that will then be mapped back to the Euclidean space.
arXiv Detail & Related papers (2021-12-21T13:09:08Z) - A Convergence Theory Towards Practical Over-parameterized Deep Neural
Networks [56.084798078072396]
We take a step towards closing the gap between theory and practice by significantly improving the known theoretical bounds on both the network width and the convergence time.
We show that convergence to a global minimum is guaranteed for networks with quadratic widths in the sample size and linear in their depth at a time logarithmic in both.
Our analysis and convergence bounds are derived via the construction of a surrogate network with fixed activation patterns that can be transformed at any time to an equivalent ReLU network of a reasonable size.
arXiv Detail & Related papers (2021-01-12T00:40:45Z) - Infinite-dimensional Folded-in-time Deep Neural Networks [0.0]
In this work, we present an infinite-dimensional generalization, which allows for a more rigorous mathematical analysis.
We also provide a functional backpropagation algorithm, which enables descent training of the weights.
arXiv Detail & Related papers (2021-01-08T11:30:50Z) - Convergence dynamics of Generative Adversarial Networks: the dual metric
flows [0.0]
We investigate convergence in the Generative Adversarial Networks used in machine learning.
We study the limit of small learning rate, and show that, similar to single network training, the GAN learning dynamics tend to vanish to some limit dynamics.
arXiv Detail & Related papers (2020-12-18T18:00:12Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.