Learning Linear Groups in Neural Networks
- URL: http://arxiv.org/abs/2305.18552v1
- Date: Mon, 29 May 2023 18:29:11 GMT
- Title: Learning Linear Groups in Neural Networks
- Authors: Emmanouil Theodosis and Karim Helwani and Demba Ba
- Abstract summary: We present a neural network architecture, Linear Group Networks (LGNs), for learning linear groups acting on the weight space of neural networks.
LGNs learn groups without any supervision or knowledge of the hidden symmetries in the data.
We demonstrate that the linear group structure depends on both the data distribution and the considered task.
- Score: 9.667333420680448
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Employing equivariance in neural networks leads to greater parameter
efficiency and improved generalization performance through the encoding of
domain knowledge in the architecture; however, the majority of existing
approaches require an a priori specification of the desired symmetries. We
present a neural network architecture, Linear Group Networks (LGNs), for
learning linear groups acting on the weight space of neural networks. Linear
groups are desirable due to their inherent interpretability, as they can be
represented as finite matrices. LGNs learn groups without any supervision or
knowledge of the hidden symmetries in the data and the groups can be mapped to
well known operations in machine learning. We use LGNs to learn groups on
multiple datasets while considering different downstream tasks; we demonstrate
that the linear group structure depends on both the data distribution and the
considered task.
Related papers
- MatrixNet: Learning over symmetry groups using learned group representations [13.19415425364914]
We propose MatrixNet, a neural network architecture that learns matrix representations of group element inputs instead of using predefined representations.
We show that MatrixNet respects group relations allowing generalization of group elements of greater word length than in the training set.
arXiv Detail & Related papers (2025-01-16T14:45:12Z) - Coding schemes in neural networks learning classification tasks [52.22978725954347]
We investigate fully-connected, wide neural networks learning classification tasks.
We show that the networks acquire strong, data-dependent features.
Surprisingly, the nature of the internal representations depends crucially on the neuronal nonlinearity.
arXiv Detail & Related papers (2024-06-24T14:50:05Z) - Feature emergence via margin maximization: case studies in algebraic
tasks [4.401622714202886]
We show that trained neural networks employ features corresponding to irreducible group-theoretic representations to perform compositions in general groups.
More generally, we hope our techniques can help to foster a deeper understanding of why neural networks adopt specific computational strategies.
arXiv Detail & Related papers (2023-11-13T18:56:33Z) - Fundamental limits of overparametrized shallow neural networks for
supervised learning [11.136777922498355]
We study a two-layer neural network trained from input-output pairs generated by a teacher network with matching architecture.
Our results come in the form of bounds relating i) the mutual information between training data and network weights, or ii) the Bayes-optimal generalization error.
arXiv Detail & Related papers (2023-07-11T08:30:50Z) - Globally Gated Deep Linear Networks [3.04585143845864]
We introduce Globally Gated Deep Linear Networks (GGDLNs) where gating units are shared among all processing units in each layer.
We derive exact equations for the generalization properties in these networks in the finite-width thermodynamic limit.
Our work is the first exact theoretical solution of learning in a family of nonlinear networks with finite width.
arXiv Detail & Related papers (2022-10-31T16:21:56Z) - FF-NSL: Feed-Forward Neural-Symbolic Learner [70.978007919101]
This paper introduces a neural-symbolic learning framework, called Feed-Forward Neural-Symbolic Learner (FF-NSL)
FF-NSL integrates state-of-the-art ILP systems based on the Answer Set semantics, with neural networks, in order to learn interpretable hypotheses from labelled unstructured data.
arXiv Detail & Related papers (2021-06-24T15:38:34Z) - Provably Training Neural Network Classifiers under Fairness Constraints [70.64045590577318]
We show that overparametrized neural networks could meet the constraints.
Key ingredient of building a fair neural network classifier is establishing no-regret analysis for neural networks.
arXiv Detail & Related papers (2020-12-30T18:46:50Z) - How Neural Networks Extrapolate: From Feedforward to Graph Neural
Networks [80.55378250013496]
We study how neural networks trained by gradient descent extrapolate what they learn outside the support of the training distribution.
Graph Neural Networks (GNNs) have shown some success in more complex tasks.
arXiv Detail & Related papers (2020-09-24T17:48:59Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z) - Neural networks adapting to datasets: learning network size and topology [77.34726150561087]
We introduce a flexible setup allowing for a neural network to learn both its size and topology during the course of a gradient-based training.
The resulting network has the structure of a graph tailored to the particular learning task and dataset.
arXiv Detail & Related papers (2020-06-22T12:46:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.