On Non-Linear operators for Geometric Deep Learning

• URL: http://arxiv.org/abs/2207.03485v1
• Date: Wed, 6 Jul 2022 06:45:33 GMT
• Title: On Non-Linear operators for Geometric Deep Learning
• Authors: Gr\'egoire Sergeant-Perthuis (LML), Jakob Maier, Joan Bruna (CIMS), Edouard Oyallon (ISIR)
• Abstract summary: We show that point-wise non-linear operators are the only universal family that commutes with any group of symmetries. It indicates that there is no universal class of non-linear operators to motivate the design of Neural Networks over the symmetries of $mathcalM$.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work studies operators mapping vector and scalar fields defined over a manifold $\mathcal{M}$, and which commute with its group of diffeomorphisms $\text{Diff}(\mathcal{M})$. We prove that in the case of scalar fields $L^p_\omega(\mathcal{M,\mathbb{R}})$, those operators correspond to point-wise non-linearities, recovering and extending known results on $\mathbb{R}^d$. In the context of Neural Networks defined over $\mathcal{M}$, it indicates that point-wise non-linear operators are the only universal family that commutes with any group of symmetries, and justifies their systematic use in combination with dedicated linear operators commuting with specific symmetries. In the case of vector fields $L^p_\omega(\mathcal{M},T\mathcal{M})$, we show that those operators are solely the scalar multiplication. It indicates that $\text{Diff}(\mathcal{M})$ is too rich and that there is no universal class of non-linear operators to motivate the design of Neural Networks over the symmetries of $\mathcal{M}$.

Related papers

• Provably learning a multi-head attention layer [55.2904547651831]
  Multi-head attention layer is one of the key components of the transformer architecture that sets it apart from traditional feed-forward models. In this work, we initiate the study of provably learning a multi-head attention layer from random examples. We prove computational lower bounds showing that in the worst case, exponential dependence on $m$ is unavoidable.
  arXiv  Detail & Related papers  (2024-02-06T15:39:09Z)
• Riemannian Langevin Monte Carlo schemes for sampling PSD matrices with fixed rank [5.0397419406319095]
  We present two explicit schemes to sample matrices from Gibbs distributions on $mathcal Sn,p_+$. We also provide examples of energy functions with explicit Gibbs distributions that allow numerical validation of these schemes.
  arXiv  Detail & Related papers  (2023-09-08T02:09:40Z)
• Deep Operator Network Approximation Rates for Lipschitz Operators [0.0]
  We establish universality and expression rate bounds for a class of neural Deep Operator Networks (DON) emulating Lipschitz (or H"older) continuous maps. The DON architecture considered uses linear encoders $mathcal E$ and decoders $mathcal D$ via (biorthogonal) Riesz bases of $mathcal X$, $mathcal Y$. Key in the proof of the present expression rate bounds is the use of either super-expressive activations.
  arXiv  Detail & Related papers  (2023-07-19T08:46:47Z)
• Quantum Current and Holographic Categorical Symmetry [62.07387569558919]
  A quantum current is defined as symmetric operators that can transport symmetry charges over an arbitrary long distance. The condition for quantum currents to be superconducting is also specified, which corresponds to condensation of anyons in one higher dimension.
  arXiv  Detail & Related papers  (2023-05-22T11:00:25Z)
• DHR bimodules of quasi-local algebras and symmetric quantum cellular automata [0.0]
  We show that for the double spin flip action $mathbbZ/2mathbbZtimes mathbbZ/2mathbbZZcurvearrowright mathbbC2otimes mathbbC2$, the group of symmetric QCA modulo symmetric finite depth circuits in 1D contains a copy of $S_3$, hence is non-abelian.
  arXiv  Detail & Related papers  (2023-03-31T18:33:07Z)
• Deep Learning Symmetries and Their Lie Groups, Algebras, and Subalgebras from First Principles [55.41644538483948]
  We design a deep-learning algorithm for the discovery and identification of the continuous group of symmetries present in a labeled dataset. We use fully connected neural networks to model the transformations symmetry and the corresponding generators. Our study also opens the door for using a machine learning approach in the mathematical study of Lie groups and their properties.
  arXiv  Detail & Related papers  (2023-01-13T16:25:25Z)
• Learning a Single Neuron with Adversarial Label Noise via Gradient Descent [50.659479930171585]
  We study a function of the form $mathbfxmapstosigma(mathbfwcdotmathbfx)$ for monotone activations. The goal of the learner is to output a hypothesis vector $mathbfw$ that $F(mathbbw)=C, epsilon$ with high probability.
  arXiv  Detail & Related papers  (2022-06-17T17:55:43Z)
• Uncertainties in Quantum Measurements: A Quantum Tomography [52.77024349608834]
  The observables associated with a quantum system $S$ form a non-commutative algebra $mathcal A_S$. It is assumed that a density matrix $rho$ can be determined from the expectation values of observables. Abelian algebras do not have inner automorphisms, so the measurement apparatus can determine mean values of observables.
  arXiv  Detail & Related papers  (2021-12-14T16:29:53Z)
• Geometric Deep Learning and Equivariant Neural Networks [0.9381376621526817]
  We survey the mathematical foundations of geometric deep learning, focusing on group equivariant and gauge equivariant neural networks. We develop gauge equivariant convolutional neural networks on arbitrary manifold $mathcalM$ using principal bundles with structure group $K$ and equivariant maps between sections of associated vector bundles. We analyze several applications of this formalism, including semantic segmentation and object detection networks.
  arXiv  Detail & Related papers  (2021-05-28T15:41:52Z)
• A Differential Geometry Perspective on Orthogonal Recurrent Models [56.09491978954866]
  We employ tools and insights from differential geometry to offer a novel perspective on orthogonal RNNs. We show that orthogonal RNNs may be viewed as optimizing in the space of divergence-free vector fields. Motivated by this observation, we study a new recurrent model, which spans the entire space of vector fields.
  arXiv  Detail & Related papers  (2021-02-18T19:39:22Z)
• Affine symmetries and neural network identifiability [0.0]
  We consider arbitrary nonlinearities with potentially complicated affine symmetries. We show that the symmetries can be used to find a rich set of networks giving rise to the same function $f$.
  arXiv  Detail & Related papers  (2020-06-21T07:09:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.