Geometric Optimisation on Manifolds with Applications to Deep Learning
- URL: http://arxiv.org/abs/2203.04794v1
- Date: Wed, 9 Mar 2022 15:20:07 GMT
- Title: Geometric Optimisation on Manifolds with Applications to Deep Learning
- Authors: Mario Lezcano-Casado
- Abstract summary: We design and implement a Python library to help the non-expert using all these powerful tools.
The algorithms implemented in this library have been designed with usability and GPU efficiency in mind.
- Score: 6.85316573653194
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We design and implement a Python library to help the non-expert using all
these powerful tools in a way that is efficient, extensible, and simple to
incorporate into the workflow of the data scientist, practitioner, and applied
researcher. The algorithms implemented in this library have been designed with
usability and GPU efficiency in mind, and they can be added to any PyTorch
model with just one extra line of code.
We showcase the effectiveness of these tools on an application of
optimisation on manifolds in the setting of time series analysis. In this
setting, orthogonal and unitary optimisation is used to constraint and
regularise recurrent models and avoid vanishing and exploding gradient
problems. The algorithms designed for GeoTorch allow us to achieve state of the
art results in the standard tests for this family of models.
We use tools from comparison geometry to give bounds on quantities that are
of interest in optimisation problems. In particular, we build on the work of
(Kaul 1976) to give explicit bounds on the norm of the second derivative of the
Riemannian exponential.
Related papers
- GloptiNets: Scalable Non-Convex Optimization with Certificates [61.50835040805378]
We present a novel approach to non-cube optimization with certificates, which handles smooth functions on the hypercube or on the torus.
By exploiting the regularity of the target function intrinsic in the decay of its spectrum, we allow at the same time to obtain precise certificates and leverage the advanced and powerful neural networks.
arXiv Detail & Related papers (2023-06-26T09:42:59Z) - EuclidNets: An Alternative Operation for Efficient Inference of Deep
Learning Models [2.715806580963474]
EuclidNet is a compression method designed to be implemented on hardware which replaces multiplication.
We show that EuclidNet is aligned with matrix multiplication and it can be used as a measure of similarity in case of convolutional layers.
arXiv Detail & Related papers (2022-12-22T15:35:42Z) - On a class of geodesically convex optimization problems solved via
Euclidean MM methods [50.428784381385164]
We show how a difference of Euclidean convexization functions can be written as a difference of different types of problems in statistics and machine learning.
Ultimately, we helps the broader broader the broader the broader the broader the work.
arXiv Detail & Related papers (2022-06-22T23:57:40Z) - Toward Learning Robust and Invariant Representations with Alignment
Regularization and Data Augmentation [76.85274970052762]
This paper is motivated by a proliferation of options of alignment regularizations.
We evaluate the performances of several popular design choices along the dimensions of robustness and invariance.
We also formally analyze the behavior of alignment regularization to complement our empirical study under assumptions we consider realistic.
arXiv Detail & Related papers (2022-06-04T04:29:19Z) - Learning to Sparsify Travelling Salesman Problem Instances [0.5985204759362747]
We use a pruning machine learning as a pre-processing step followed by an exact Programming approach to sparsify the travelling salesman problem.
Our learning approach requires very little training data and is amenable to mathematical analysis.
arXiv Detail & Related papers (2021-04-19T14:35:14Z) - Self Normalizing Flows [65.73510214694987]
We propose a flexible framework for training normalizing flows by replacing expensive terms in the gradient by learned approximate inverses at each layer.
This reduces the computational complexity of each layer's exact update from $mathcalO(D3)$ to $mathcalO(D2)$.
We show experimentally that such models are remarkably stable and optimize to similar data likelihood values as their exact gradient counterparts.
arXiv Detail & Related papers (2020-11-14T09:51:51Z) - Kernel methods through the roof: handling billions of points efficiently [94.31450736250918]
Kernel methods provide an elegant and principled approach to nonparametric learning, but so far could hardly be used in large scale problems.
Recent advances have shown the benefits of a number of algorithmic ideas, for example combining optimization, numerical linear algebra and random projections.
Here, we push these efforts further to develop and test a solver that takes full advantage of GPU hardware.
arXiv Detail & Related papers (2020-06-18T08:16:25Z) - Physarum Powered Differentiable Linear Programming Layers and
Applications [48.77235931652611]
We propose an efficient and differentiable solver for general linear programming problems.
We show the use of our solver in a video segmentation task and meta-learning for few-shot learning.
arXiv Detail & Related papers (2020-04-30T01:50:37Z) - An Advance on Variable Elimination with Applications to Tensor-Based
Computation [11.358487655918676]
We present new results on the classical algorithm of variable elimination, which underlies many algorithms including for probabilistic inference.
The results relate to exploiting functional dependencies, allowing one to perform inference and learning efficiently on models that have very large treewidth.
arXiv Detail & Related papers (2020-02-21T14:17:44Z) - PHOTONAI -- A Python API for Rapid Machine Learning Model Development [2.414341608751139]
PHOTONAI is a high-level Python API designed to simplify and accelerate machine learning model development.
It functions as a unifying framework allowing the user to easily access and combine algorithms from different toolboxes into custom algorithm sequences.
arXiv Detail & Related papers (2020-02-13T10:33:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.