Convex Relaxation for Solving Large-Margin Classifiers in Hyperbolic Space

• URL: http://arxiv.org/abs/2405.17198v1
• Date: Mon, 27 May 2024 14:19:53 GMT
• Title: Convex Relaxation for Solving Large-Margin Classifiers in Hyperbolic Space
• Authors: Sheng Yang, Peihan Liu, Cengiz Pehlevan,
• Abstract summary: Hyperbolic spaces have been recognized for their outstanding performance in handling data. Previous attempts to solve this problem using a gradient descent approach have failed. In this paper we show how we can effectively approximate the optima using sparse moment of relaxations.
• Score: 29.564717407207528
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Hyperbolic spaces have increasingly been recognized for their outstanding performance in handling data with inherent hierarchical structures compared to their Euclidean counterparts. However, learning in hyperbolic spaces poses significant challenges. In particular, extending support vector machines to hyperbolic spaces is in general a constrained non-convex optimization problem. Previous and popular attempts to solve hyperbolic SVMs, primarily using projected gradient descent, are generally sensitive to hyperparameters and initializations, often leading to suboptimal solutions. In this work, by first rewriting the problem into a polynomial optimization, we apply semidefinite relaxation and sparse moment-sum-of-squares relaxation to effectively approximate the optima. From extensive empirical experiments, these methods are shown to perform better than the projected gradient descent approach.

Related papers

• Independently-Normalized SGD for Generalized-Smooth Nonconvex Optimization [19.000530691874516]
  We show that many non machine learning problems meet that kind of condition that extends beyond traditional non-smoothepseps. We propose an independently-normalized gradient descent algorithm, which leverages independent sampling and normalization.
  arXiv  Detail & Related papers  (2024-10-17T21:52:00Z)
• Low-Rank Extragradient Methods for Scalable Semidefinite Optimization [0.0]
  We focus on high-dimensional and plausible settings in which the problem admits a low-rank solution. We provide several theoretical results proving that, under these circumstances, the well-known Extragradient method converges to a solution of the constrained optimization problem.
  arXiv  Detail & Related papers  (2024-02-14T10:48:00Z)
• Fast hyperboloid decision tree algorithms [0.6656737591902598]
  We present hyperDT, a novel extension of decision tree algorithms into hyperbolic space. Our approach is conceptually straightforward and maintains constant-time decision complexity. Building upon hyperDT we introduce hyperRF, a hyperbolic random forest model.
  arXiv  Detail & Related papers  (2023-10-20T22:31:10Z)
• Horospherical Decision Boundaries for Large Margin Classification in Hyperbolic Space [8.901073744693317]
  We propose a novel convex descent problem that can be optimized using a large margin setting. We present several experiments depicting the competitive performance of our algorithms in comparison to SOTA.
  arXiv  Detail & Related papers  (2023-02-14T03:26:19Z)
• HRCF: Enhancing Collaborative Filtering via Hyperbolic Geometric Regularization [52.369435664689995]
  We introduce a textitHyperbolic Regularization powered Collaborative Filtering (HRCF) and design a geometric-aware hyperbolic regularizer. Specifically, the proposal boosts optimization procedure via the root alignment and origin-aware penalty. Our proposal is able to tackle the over-smoothing problem caused by hyperbolic aggregation and also brings the models a better discriminative ability.
  arXiv  Detail & Related papers  (2022-04-18T06:11:44Z)
• Faster Algorithm and Sharper Analysis for Constrained Markov Decision Process [56.55075925645864]
  The problem of constrained decision process (CMDP) is investigated, where an agent aims to maximize the expected accumulated discounted reward subject to multiple constraints. A new utilities-dual convex approach is proposed with novel integration of three ingredients: regularized policy, dual regularizer, and Nesterov's gradient descent dual. This is the first demonstration that nonconcave CMDP problems can attain the lower bound of $mathcal O (1/epsilon)$ for all complexity optimization subject to convex constraints.
  arXiv  Detail & Related papers  (2021-10-20T02:57:21Z)
• Optimization on manifolds: A symplectic approach [127.54402681305629]
  We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
  arXiv  Detail & Related papers  (2021-07-23T13:43:34Z)
• Implicit differentiation for fast hyperparameter selection in non-smooth convex learning [87.60600646105696]
  We study first-order methods when the inner optimization problem is convex but non-smooth. We show that the forward-mode differentiation of proximal gradient descent and proximal coordinate descent yield sequences of Jacobians converging toward the exact Jacobian.
  arXiv  Detail & Related papers  (2021-05-04T17:31:28Z)
• Convergence Properties of Stochastic Hypergradients [38.64355126221992]
  We study approximation schemes for the hypergradient, which are important when the lower-level problem is empirical risk on a large dataset. We provide numerical experiments to support our theoretical analysis and to show the advantage of using hypergradients in practice.
  arXiv  Detail & Related papers  (2020-11-13T20:50:36Z)
• Efficient Methods for Structured Nonconvex-Nonconcave Min-Max Optimization [98.0595480384208]
  We propose a generalization extraient spaces which converges to a stationary point. The algorithm applies not only to general $p$-normed spaces, but also to general $p$-dimensional vector spaces.
  arXiv  Detail & Related papers  (2020-10-31T21:35:42Z)
• Implicit differentiation of Lasso-type models for hyperparameter optimization [82.73138686390514]
  We introduce an efficient implicit differentiation algorithm, without matrix inversion, tailored for Lasso-type problems. Our approach scales to high-dimensional data by leveraging the sparsity of the solutions.
  arXiv  Detail & Related papers  (2020-02-20T18:43:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.