A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold

• URL: http://arxiv.org/abs/2302.08210v1
• Date: Thu, 16 Feb 2023 10:50:15 GMT
• Title: A Survey of Geometric Optimization for Deep Learning: From Euclidean Space to Riemannian Manifold
• Authors: Yanhong Fei, Xian Wei, Yingjie Liu, Zhengyu Li, Mingsong Chen
• Abstract summary: Deep Learning (DL) has achieved success in complex Artificial Intelligence (AI) tasks, but it suffers from various notorious problems. This article presents a comprehensive survey of applying geometric optimization in DL. It investigates the application of geometric optimization in different DL networks in various AI tasks, e.g., convolution neural network, recurrent neural network, transfer learning, and optimal transport.
• Score: 7.737713458418288
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Deep Learning (DL) has achieved success in complex Artificial Intelligence (AI) tasks, it suffers from various notorious problems (e.g., feature redundancy, and vanishing or exploding gradients), since updating parameters in Euclidean space cannot fully exploit the geometric structure of the solution space. As a promising alternative solution, Riemannian-based DL uses geometric optimization to update parameters on Riemannian manifolds and can leverage the underlying geometric information. Accordingly, this article presents a comprehensive survey of applying geometric optimization in DL. At first, this article introduces the basic procedure of the geometric optimization, including various geometric optimizers and some concepts of Riemannian manifold. Subsequently, this article investigates the application of geometric optimization in different DL networks in various AI tasks, e.g., convolution neural network, recurrent neural network, transfer learning, and optimal transport. Additionally, typical public toolboxes that implement optimization on manifold are also discussed. Finally, this article makes a performance comparison between different deep geometric optimization methods under image recognition scenarios.

Related papers

• Randomized Geometric Algebra Methods for Convex Neural Networks [45.318490912354825]
  We introduce randomized algorithms to Clifford's Geometric Algebra, generalizing randomized linear algebra to hypercomplex vector spaces. This novel approach has many implications in machine learning, including training neural networks to global optimality via convex optimization.
  arXiv  Detail & Related papers  (2024-06-04T22:22:39Z)
• Riemannian Self-Attention Mechanism for SPD Networks [34.794770395408335]
  An SPD manifold self-attention mechanism (SMSA) is proposed in this paper. An SMSA-based geometric learning module (SMSA-GL) is designed for the sake of improving the discrimination of structured representations.
  arXiv  Detail & Related papers  (2023-11-28T12:34:46Z)
• Physics-informed neural networks for transformed geometries and manifolds [0.0]
  We propose a novel method for integrating geometric transformations within PINNs to robustly accommodate geometric variations. We demonstrate the enhanced flexibility over traditional PINNs, especially under geometric variations. The proposed framework presents an outlook for training deep neural operators over parametrized geometries.
  arXiv  Detail & Related papers  (2023-11-27T15:47:33Z)
• Optimizing Solution-Samplers for Combinatorial Problems: The Landscape of Policy-Gradient Methods [52.0617030129699]
  We introduce a novel theoretical framework for analyzing the effectiveness of DeepMatching Networks and Reinforcement Learning methods. Our main contribution holds for a broad class of problems including Max-and Min-Cut, Max-$k$-Bipartite-Bi, Maximum-Weight-Bipartite-Bi, and Traveling Salesman Problem. As a byproduct of our analysis we introduce a novel regularization process over vanilla descent and provide theoretical and experimental evidence that it helps address vanishing-gradient issues and escape bad stationary points.
  arXiv  Detail & Related papers  (2023-10-08T23:39:38Z)
• Adaptive Log-Euclidean Metrics for SPD Matrix Learning [73.12655932115881]
  We propose Adaptive Log-Euclidean Metrics (ALEMs), which extend the widely used Log-Euclidean Metric (LEM) The experimental and theoretical results demonstrate the merit of the proposed metrics in improving the performance of SPD neural networks.
  arXiv  Detail & Related papers  (2023-03-26T18:31:52Z)
• On a class of geodesically convex optimization problems solved via Euclidean MM methods [50.428784381385164]
  We show how a difference of Euclidean convexization functions can be written as a difference of different types of problems in statistics and machine learning. Ultimately, we helps the broader broader the broader the broader the broader the work.
  arXiv  Detail & Related papers  (2022-06-22T23:57:40Z)
• Geometric Methods for Sampling, Optimisation, Inference and Adaptive Agents [102.42623636238399]
  We identify fundamental geometric structures that underlie the problems of sampling, optimisation, inference and adaptive decision-making. We derive algorithms that exploit these geometric structures to solve these problems efficiently.
  arXiv  Detail & Related papers  (2022-03-20T16:23:17Z)
• On Geometric Connections of Embedded and Quotient Geometries in Riemannian Fixed-rank Matrix Optimization [5.876141028192136]
  This paper proposes a general procedure for establishing the geometric landscape connections of a Riemannian optimization problem under the embedded and quotient geometries. We observe an algorithmic connection between two geometries with some specific Riemannian metrics in fixed-rank matrix optimization. Results provide a few new theoretical insights to unanswered questions in the literature.
  arXiv  Detail & Related papers  (2021-10-23T03:13:56Z)
• Hybrid neural network reduced order modelling for turbulent flows with geometric parameters [0.0]
  This paper introduces a new technique mixing up a classical Galerkin-projection approach together with a data-driven method to obtain a versatile and accurate algorithm for the resolution of geometrically parametrized incompressible turbulent Navier-Stokes problems. The effectiveness of this procedure is demonstrated on two different test cases: a classical academic back step problem and a shape deformation Ahmed body application.
  arXiv  Detail & Related papers  (2021-07-20T16:06:18Z)
• ResNet-LDDMM: Advancing the LDDMM Framework Using Deep Residual Networks [86.37110868126548]
  In this work, we make use of deep residual neural networks to solve the non-stationary ODE (flow equation) based on a Euler's discretization scheme. We illustrate these ideas on diverse registration problems of 3D shapes under complex topology-preserving transformations.
  arXiv  Detail & Related papers  (2021-02-16T04:07:13Z)
• Learning to Guide Random Search [111.71167792453473]
  We consider derivative-free optimization of a high-dimensional function that lies on a latent low-dimensional manifold. We develop an online learning approach that learns this manifold while performing the optimization. We empirically evaluate the method on continuous optimization benchmarks and high-dimensional continuous control problems.
  arXiv  Detail & Related papers  (2020-04-25T19:21:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.