Variational Optimization on Lie Groups, with Examples of Leading (Generalized) Eigenvalue Problems

• URL: http://arxiv.org/abs/2001.10006v1
• Date: Mon, 27 Jan 2020 19:00:03 GMT
• Title: Variational Optimization on Lie Groups, with Examples of Leading (Generalized) Eigenvalue Problems
• Authors: Molei Tao, Tomoki Ohsawa
• Abstract summary: The article considers smooth optimization of functions on Lie groups. By generalizing NAG variational principle in vector space to Lie groups, continuous Lie-NAG dynamics which are guaranteed to converge to local optimum are obtained.
• Score: 10.203602318836444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The article considers smooth optimization of functions on Lie groups. By generalizing NAG variational principle in vector space (Wibisono et al., 2016) to Lie groups, continuous Lie-NAG dynamics which are guaranteed to converge to local optimum are obtained. They correspond to momentum versions of gradient flow on Lie groups. A particular case of $\mathsf{SO}(n)$ is then studied in details, with objective functions corresponding to leading Generalized EigenValue problems: the Lie-NAG dynamics are first made explicit in coordinates, and then discretized in structure preserving fashions, resulting in optimization algorithms with faithful energy behavior (due to conformal symplecticity) and exactly remaining on the Lie group. Stochastic gradient versions are also investigated. Numerical experiments on both synthetic data and practical problem (LDA for MNIST) demonstrate the effectiveness of the proposed methods as optimization algorithms ($not$ as a classification method).

Related papers

• A Fresh Look at Generalized Category Discovery through Non-negative Matrix Factorization [83.12938977698988]
  Generalized Category Discovery (GCD) aims to classify both base and novel images using labeled base data. Current approaches inadequately address the intrinsic optimization of the co-occurrence matrix $barA$ based on cosine similarity. We propose a Non-Negative Generalized Category Discovery (NN-GCD) framework to address these deficiencies.
  arXiv  Detail & Related papers  (2024-10-29T07:24:11Z)
• Quantitative Convergences of Lie Group Momentum Optimizers [21.76159063788814]
  This article investigates two types of discretization, Lie Heavy-Ball and Lie NAG-SC, which is newly proposed. Lie Heavy-Ball and Lie NAG-SC are computationally cheaper and easier to implement, thanks to their utilization of group structure.
  arXiv  Detail & Related papers  (2024-05-30T18:01:14Z)
• Stochastic Hessian Fittings with Lie Groups [6.626539885456148]
  Hessian fitting as an optimization problem is strongly convex under mild conditions with a specific yet general enough Lie group. This discovery turns Hessian fitting into a well behaved optimization problem, and facilitates the designs of highly efficient and elegant Lie group sparse preconditioner fitting methods for large scale optimizations.
  arXiv  Detail & Related papers  (2024-02-19T06:00:35Z)
• Optimal Potential Shaping on SE(3) via Neural ODEs on Lie Groups [4.595636071139437]
  We rephrase dynamic systems as neural ordinary differential equations (neural ODEs) A gradient descent optimization algorithm is presented to tackle the optimization numerically. In an extensive example, optimal potential energy shaping for control of a rigid body is treated.
  arXiv  Detail & Related papers  (2024-01-25T16:26:44Z)
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD [73.55632827932101]
  We optimize the information-theoretical generalization bound by manipulating the noise structure in SGLD. We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance.
  arXiv  Detail & Related papers  (2021-10-26T15:02:27Z)
• Generalization Properties of Stochastic Optimizers via Trajectory Analysis [48.38493838310503]
  We show that both the Fernique-Talagrand functional and the local powerlaw are predictive of generalization performance. We show that both our Fernique-Talagrand functional and the local powerlaw are predictive of generalization performance.
  arXiv  Detail & Related papers  (2021-08-02T10:58:32Z)
• LieTransformer: Equivariant self-attention for Lie Groups [49.9625160479096]
  Group equivariant neural networks are used as building blocks of group invariant neural networks. We extend the scope of the literature to self-attention, that is emerging as a prominent building block of deep learning models. We propose the LieTransformer, an architecture composed of LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
  arXiv  Detail & Related papers  (2020-12-20T11:02:49Z)
• Stochastic batch size for adaptive regularization in deep network optimization [63.68104397173262]
  We propose a first-order optimization algorithm incorporating adaptive regularization applicable to machine learning problems in deep learning framework. We empirically demonstrate the effectiveness of our algorithm using an image classification task based on conventional network models applied to commonly used benchmark datasets.
  arXiv  Detail & Related papers  (2020-04-14T07:54:53Z)
• Stochastic Flows and Geometric Optimization on the Orthogonal Group [52.50121190744979]
  We present a new class of geometrically-driven optimization algorithms on the orthogonal group $O(d)$. We show that our methods can be applied in various fields of machine learning including deep, convolutional and recurrent neural networks, reinforcement learning, flows and metric learning.
  arXiv  Detail & Related papers  (2020-03-30T15:37:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.