A Functional Perspective on Learning Symmetric Functions with Neural
Networks
- URL: http://arxiv.org/abs/2008.06952v4
- Date: Mon, 10 Oct 2022 14:55:01 GMT
- Title: A Functional Perspective on Learning Symmetric Functions with Neural
Networks
- Authors: Aaron Zweig, Joan Bruna
- Abstract summary: We study the learning and representation of neural networks defined on measures.
We establish approximation and generalization bounds under different choices of regularization.
The resulting models can be learned efficiently and enjoy generalization guarantees that extend across input sizes.
- Score: 48.80300074254758
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Symmetric functions, which take as input an unordered, fixed-size set, are
known to be universally representable by neural networks that enforce
permutation invariance. These architectures only give guarantees for fixed
input sizes, yet in many practical applications, including point clouds and
particle physics, a relevant notion of generalization should include varying
the input size. In this work we treat symmetric functions (of any size) as
functions over probability measures, and study the learning and representation
of neural networks defined on measures. By focusing on shallow architectures,
we establish approximation and generalization bounds under different choices of
regularization (such as RKHS and variation norms), that capture a hierarchy of
functional spaces with increasing degree of non-linear learning. The resulting
models can be learned efficiently and enjoy generalization guarantees that
extend across input sizes, as we verify empirically.
Related papers
- Deep Neural Networks are Adaptive to Function Regularity and Data Distribution in Approximation and Estimation [8.284464143581546]
We study how deep neural networks can adapt to different regularity in functions across different locations and scales.
Our results show that deep neural networks are adaptive to different regularity of functions and nonuniform data distributions.
arXiv Detail & Related papers (2024-06-08T02:01:50Z) - Geometry-induced Implicit Regularization in Deep ReLU Neural Networks [0.0]
Implicit regularization phenomena, which are still not well understood, occur during optimization.
We study the geometry of the output set as parameters vary.
We prove that the batch functional dimension is almost surely determined by the activation patterns in the hidden layers.
arXiv Detail & Related papers (2024-02-13T07:49:57Z) - Function-Space Regularization in Neural Networks: A Probabilistic
Perspective [51.133793272222874]
We show that we can derive a well-motivated regularization technique that allows explicitly encoding information about desired predictive functions into neural network training.
We evaluate the utility of this regularization technique empirically and demonstrate that the proposed method leads to near-perfect semantic shift detection and highly-calibrated predictive uncertainty estimates.
arXiv Detail & Related papers (2023-12-28T17:50:56Z) - Any-dimensional equivariant neural networks [1.4469725791865984]
Traditional supervised learning aims to learn an unknown mapping by fitting a function to a set of input-output pairs with a fixed dimension.
We leverage a newly-discovered phenomenon in algebraic topology, called representation stability, to define equivariant neural networks that can be trained with data in a fixed dimension.
arXiv Detail & Related papers (2023-06-10T00:55:38Z) - Permutation Equivariant Neural Functionals [92.0667671999604]
This work studies the design of neural networks that can process the weights or gradients of other neural networks.
We focus on the permutation symmetries that arise in the weights of deep feedforward networks because hidden layer neurons have no inherent order.
In our experiments, we find that permutation equivariant neural functionals are effective on a diverse set of tasks.
arXiv Detail & Related papers (2023-02-27T18:52:38Z) - Equivariance with Learned Canonicalization Functions [77.32483958400282]
We show that learning a small neural network to perform canonicalization is better than using predefineds.
Our experiments show that learning the canonicalization function is competitive with existing techniques for learning equivariant functions across many tasks.
arXiv Detail & Related papers (2022-11-11T21:58:15Z) - Deep Neural Network Approximation of Invariant Functions through
Dynamical Systems [15.716533830931766]
We study the approximation of functions which are invariant with respect to certain permutations of the input indices using flow maps of dynamical systems.
Such invariant functions includes the much studied translation-invariant ones involving image tasks, but also encompasses many permutation-invariant functions that finds emerging applications in science and engineering.
arXiv Detail & Related papers (2022-08-18T08:36:16Z) - Stabilizing Q-learning with Linear Architectures for Provably Efficient
Learning [53.17258888552998]
This work proposes an exploration variant of the basic $Q$-learning protocol with linear function approximation.
We show that the performance of the algorithm degrades very gracefully under a novel and more permissive notion of approximation error.
arXiv Detail & Related papers (2022-06-01T23:26:51Z) - Frame Averaging for Invariant and Equivariant Network Design [50.87023773850824]
We introduce Frame Averaging (FA), a framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types.
We show that FA-based models have maximal expressive power in a broad setting.
We propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs.
arXiv Detail & Related papers (2021-10-07T11:05:23Z) - Non-Euclidean Universal Approximation [4.18804572788063]
Modifications to a neural network's input and output layers are often required to accommodate the specificities of most practical learning tasks.
We present general conditions describing feature and readout maps that preserve an architecture's ability to approximate any continuous functions uniformly on compacts.
arXiv Detail & Related papers (2020-06-03T15:38:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.