Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks

• URL: http://arxiv.org/abs/2310.03530v2
• Date: Mon, 13 Nov 2023 11:56:32 GMT
• Title: Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks
• Authors: Sho Sonoda, Hideyuki Ishi, Isao Ishikawa, Masahiro Ikeda
• Abstract summary: We present a systematic method to induce a generalized neural network and its right inverse operator, called the ridgelet transform. Since the ridgelet transform is an inverse, it can describe the arrangement of parameters for the network to represent a target function. We present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks.
• Score: 14.45619075342763
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The symmetry and geometry of input data are considered to be encoded in the internal data representation inside the neural network, but the specific encoding rule has been less investigated. In this study, we present a systematic method to induce a generalized neural network and its right inverse operator, called the ridgelet transform, from a joint group invariant function on the data-parameter domain. Since the ridgelet transform is an inverse, (1) it can describe the arrangement of parameters for the network to represent a target function, which is understood as the encoding rule, and (2) it implies the universality of the network. Based on the group representation theory, we present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks, for example, the original ridgelet transform, formal deep networks, and the dual voice transform. Since traditional universality theorems were demonstrated based on functional analysis, this study sheds light on the group theoretic aspect of the approximation theory, connecting geometric deep learning to abstract harmonic analysis.

Related papers

• Relative Representations: Topological and Geometric Perspectives [53.88896255693922]
  Relative representations are an established approach to zero-shot model stitching. We introduce a normalization procedure in the relative transformation, resulting in invariance to non-isotropic rescalings and permutations. Second, we propose to deploy topological densification when fine-tuning relative representations, a topological regularization loss encouraging clustering within classes.
  arXiv  Detail & Related papers  (2024-09-17T08:09:22Z)
• Affine Invariance in Continuous-Domain Convolutional Neural Networks [6.019182604573028]
  This research studies affine invariance on continuous-domain convolutional neural networks. We introduce a new criterion to assess the similarity of two input signals under affine transformations. Our research could eventually extend the scope of geometrical transformations that practical deep-learning pipelines can handle.
  arXiv  Detail & Related papers  (2023-11-13T14:17:57Z)
• Orthogonal Transforms in Neural Networks Amount to Effective Regularization [0.0]
  We consider applications of neural networks in nonlinear system identification. We show that such a structure is a universal approximator. We empirically show in particular, that such a structure, using the Fourier transform, outperforms equivalent models without orthogonality support.
  arXiv  Detail & Related papers  (2023-05-10T17:52:33Z)
• Universality of group convolutional neural networks based on ridgelet analysis on groups [10.05944106581306]
  We investigate the approximation property of group convolutional neural networks (GCNNs) based on the ridgelet theory. We formulate a versatile GCNN as a nonlinear mapping between group representations.
  arXiv  Detail & Related papers  (2022-05-30T02:52:22Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• The Sample Complexity of One-Hidden-Layer Neural Networks [57.6421258363243]
  We study a class of scalar-valued one-hidden-layer networks, and inputs bounded in Euclidean norm. We prove that controlling the spectral norm of the hidden layer weight matrix is insufficient to get uniform convergence guarantees. We analyze two important settings where a mere spectral norm control turns out to be sufficient.
  arXiv  Detail & Related papers  (2022-02-13T07:12:02Z)
• Revisiting Transformation Invariant Geometric Deep Learning: Are Initial Representations All You Need? [80.86819657126041]
  We show that transformation-invariant and distance-preserving initial representations are sufficient to achieve transformation invariance. Specifically, we realize transformation-invariant and distance-preserving initial point representations by modifying multi-dimensional scaling. We prove that TinvNN can strictly guarantee transformation invariance, being general and flexible enough to be combined with the existing neural networks.
  arXiv  Detail & Related papers  (2021-12-23T03:52:33Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Self-Supervised Graph Representation Learning via Topology Transformations [61.870882736758624]
  We present the Topology Transformation Equivariant Representation learning, a general paradigm of self-supervised learning for node representations of graph data. In experiments, we apply the proposed model to the downstream node and graph classification tasks, and results show that the proposed method outperforms the state-of-the-art unsupervised approaches.
  arXiv  Detail & Related papers  (2021-05-25T06:11:03Z)
• Lattice gauge equivariant convolutional neural networks [0.0]
  We propose Lattice gauge equivariant Convolutional Neural Networks (L-CNNs) for generic machine learning applications. We show that L-CNNs can learn and generalize gauge invariant quantities that traditional convolutional neural networks are incapable of finding.
  arXiv  Detail & Related papers  (2020-12-23T19:00:01Z)
• A Computationally Efficient Neural Network Invariant to the Action of Symmetry Subgroups [12.654871396334668]
  A new $G$-invariant transformation module produces a $G$-invariant latent representation of the input data. This latent representation is then processed with a multi-layer perceptron in the network. We prove the universality of the proposed architecture, discuss its properties and highlight its computational and memory efficiency.
  arXiv  Detail & Related papers  (2020-02-18T12:50:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.