Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
- URL: http://arxiv.org/abs/2104.13478v1
- Date: Tue, 27 Apr 2021 21:09:51 GMT
- Title: Geometric Deep Learning: Grids, Groups, Graphs, Geodesics, and Gauges
- Authors: Michael M. Bronstein, Joan Bruna, Taco Cohen, Petar Veli\v{c}kovi\'c
- Abstract summary: The last decade has witnessed an experimental revolution in data science and machine learning, epitomised by deep learning methods.
This text is concerned with exposing pre-defined regularities through unified geometric principles.
It provides a common mathematical framework to study the most successful neural network architectures, such as CNNs, RNNs, GNNs, and Transformers.
- Score: 50.22269760171131
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The last decade has witnessed an experimental revolution in data science and
machine learning, epitomised by deep learning methods. Indeed, many
high-dimensional learning tasks previously thought to be beyond reach -- such
as computer vision, playing Go, or protein folding -- are in fact feasible with
appropriate computational scale. Remarkably, the essence of deep learning is
built from two simple algorithmic principles: first, the notion of
representation or feature learning, whereby adapted, often hierarchical,
features capture the appropriate notion of regularity for each task, and
second, learning by local gradient-descent type methods, typically implemented
as backpropagation.
While learning generic functions in high dimensions is a cursed estimation
problem, most tasks of interest are not generic, and come with essential
pre-defined regularities arising from the underlying low-dimensionality and
structure of the physical world. This text is concerned with exposing these
regularities through unified geometric principles that can be applied
throughout a wide spectrum of applications.
Such a 'geometric unification' endeavour, in the spirit of Felix Klein's
Erlangen Program, serves a dual purpose: on one hand, it provides a common
mathematical framework to study the most successful neural network
architectures, such as CNNs, RNNs, GNNs, and Transformers. On the other hand,
it gives a constructive procedure to incorporate prior physical knowledge into
neural architectures and provide principled way to build future architectures
yet to be invented.
Related papers
- Reasoning Algorithmically in Graph Neural Networks [1.8130068086063336]
We aim to integrate the structured and rule-based reasoning of algorithms with adaptive learning capabilities of neural networks.
This dissertation provides theoretical and practical contributions to this area of research.
arXiv Detail & Related papers (2024-02-21T12:16:51Z) - Training Neural Networks with Internal State, Unconstrained
Connectivity, and Discrete Activations [66.53734987585244]
True intelligence may require the ability of a machine learning model to manage internal state.
We show that we have not yet discovered the most effective algorithms for training such models.
We present one attempt to design such a training algorithm, applied to an architecture with binary activations and only a single matrix of weights.
arXiv Detail & Related papers (2023-12-22T01:19:08Z) - Riemannian Residual Neural Networks [58.925132597945634]
We show how to extend the residual neural network (ResNet)
ResNets have become ubiquitous in machine learning due to their beneficial learning properties, excellent empirical results, and easy-to-incorporate nature when building varied neural networks.
arXiv Detail & Related papers (2023-10-16T02:12:32Z) - Quasi-orthogonality and intrinsic dimensions as measures of learning and
generalisation [55.80128181112308]
We show that dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as network's performance discriminants.
Our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces.
arXiv Detail & Related papers (2022-03-30T21:47:32Z) - Intrinsic Dimension, Persistent Homology and Generalization in Neural
Networks [19.99615698375829]
We show that generalization error can be equivalently bounded in terms of a notion called the 'persistent homology dimension' (PHD)
We develop an efficient algorithm to estimate PHD in the scale of modern deep neural networks.
Our experiments show that the proposed approach can efficiently compute a network's intrinsic dimension in a variety of settings.
arXiv Detail & Related papers (2021-11-25T17:06:15Z) - A neural anisotropic view of underspecification in deep learning [60.119023683371736]
We show that the way neural networks handle the underspecification of problems is highly dependent on the data representation.
Our results highlight that understanding the architectural inductive bias in deep learning is fundamental to address the fairness, robustness, and generalization of these systems.
arXiv Detail & Related papers (2021-04-29T14:31:09Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z) - Structure preserving deep learning [1.2263454117570958]
deep learning has risen to the foreground as a topic of massive interest.
There are multiple challenging mathematical problems involved in applying deep learning.
A growing effort to mathematically understand the structure in existing deep learning methods.
arXiv Detail & Related papers (2020-06-05T10:59:09Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.