Geometry of Polynomial Neural Networks
- URL: http://arxiv.org/abs/2402.00949v1
- Date: Thu, 1 Feb 2024 19:06:06 GMT
- Title: Geometry of Polynomial Neural Networks
- Authors: Kaie Kubjas, Jiayi Li, Maximilian Wiesmann
- Abstract summary: We study the expressivity and learning process for neural networks (PNNs) with monomial activation functions.
These theoretical results are accompanied by experiments.
- Score: 3.9318191265352196
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: We study the expressivity and learning process for polynomial neural networks
(PNNs) with monomial activation functions. The weights of the network
parametrize the neuromanifold. In this paper, we study certain neuromanifolds
using tools from algebraic geometry: we give explicit descriptions as
semialgebraic sets and characterize their Zariski closures, called
neurovarieties. We study their dimension and associate an algebraic degree, the
learning degree, to the neurovariety. The dimension serves as a geometric
measure for the expressivity of the network, the learning degree is a measure
for the complexity of training the network and provides upper bounds on the
number of learnable functions. These theoretical results are accompanied with
experiments.
Related papers
- Tropical Expressivity of Neural Networks [0.0]
A particular quantity that has been actively studied in the field of deep learning is the number of linear regions.
To study and evaluate information capacity and expressivity, we work in the setting of tropical geometry.
Our work builds on and expands this connection to capitalize on the rich theory of tropical geometry.
arXiv Detail & Related papers (2024-05-30T15:45:03Z) - A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes [49.32130498861987]
We study the case of non-differentiable activation functions, such as ReLU.
Two recent works introduced a geometric framework to study neural networks.
We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.
arXiv Detail & Related papers (2024-04-09T08:11:46Z) - Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
We propose to represent neural networks as computational graphs of parameters.
Our approach enables a single model to encode neural computational graphs with diverse architectures.
We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
arXiv Detail & Related papers (2024-03-18T18:01:01Z) - Universal Approximation Theorem for Vector- and Hypercomplex-Valued
Neural Networks [0.4143603294943439]
The universal approximation theorem states that a neural network with one hidden layer can approximate continuous functions on compact sets.
It is valid for real-valued neural networks and some hypercomplex-valued neural networks.
arXiv Detail & Related papers (2024-01-04T13:56:13Z) - Randomly Weighted Neuromodulation in Neural Networks Facilitates
Learning of Manifolds Common Across Tasks [1.9580473532948401]
Geometric Sensitive Hashing functions are neural network models that learn class-specific manifold geometry in supervised learning.
We show that a randomly weighted neural network with a neuromodulation system can realize this function.
arXiv Detail & Related papers (2023-11-17T15:22:59Z) - Riemannian Residual Neural Networks [58.925132597945634]
We show how to extend the residual neural network (ResNet)
ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
arXiv Detail & Related papers (2023-10-16T02:12:32Z) - Permutation Equivariant Neural Functionals [92.0667671999604]
This work studies the design of neural networks that can process the weights or gradients of other neural networks.
We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order.
In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
arXiv Detail & Related papers (2023-02-27T18:52:38Z) - Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a
Polynomial Net Study [55.12108376616355]
The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp)
In this work, we derive the finite-width-K formulation for a special class of NNs-Hp, i.e., neural networks.
We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK.
arXiv Detail & Related papers (2022-09-16T06:36:06Z) - Neural population geometry: An approach for understanding biological and
artificial neural networks [3.4809730725241605]
We review examples of geometrical approaches providing insight into the function of biological and artificial neural networks.
Neural population geometry has the potential to unify our understanding of structure and function in biological and artificial neural networks.
arXiv Detail & Related papers (2021-04-14T18:10:34Z) - Stability of Algebraic Neural Networks to Small Perturbations [179.55535781816343]
Algebraic neural networks (AlgNNs) are composed of a cascade of layers each one associated to and algebraic signal model.
We show how any architecture that uses a formal notion of convolution can be stable beyond particular choices of the shift operator.
arXiv Detail & Related papers (2020-10-22T09:10:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.