The Numerical Stability of Hyperbolic Representation Learning

• URL: http://arxiv.org/abs/2211.00181v3
• Date: Wed, 28 Jun 2023 02:54:30 GMT
• Title: The Numerical Stability of Hyperbolic Representation Learning
• Authors: Gal Mishne, Zhengchao Wan, Yusu Wang, Sheng Yang
• Abstract summary: We analyze the limitations of two popular models for the hyperbolic space, namely, the Poincar'e ball and the Lorentz model. We extend this Euclidean parametrization to hyperbolic hyperplanes and exhibit its ability to improve the performance of hyperbolic SVM.
• Score: 36.32817250000654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given the exponential growth of the volume of the ball w.r.t. its radius, the hyperbolic space is capable of embedding trees with arbitrarily small distortion and hence has received wide attention for representing hierarchical datasets. However, this exponential growth property comes at a price of numerical instability such that training hyperbolic learning models will sometimes lead to catastrophic NaN problems, encountering unrepresentable values in floating point arithmetic. In this work, we carefully analyze the limitation of two popular models for the hyperbolic space, namely, the Poincar\'e ball and the Lorentz model. We first show that, under the 64 bit arithmetic system, the Poincar\'e ball has a relatively larger capacity than the Lorentz model for correctly representing points. Then, we theoretically validate the superiority of the Lorentz model over the Poincar\'e ball from the perspective of optimization. Given the numerical limitations of both models, we identify one Euclidean parametrization of the hyperbolic space which can alleviate these limitations. We further extend this Euclidean parametrization to hyperbolic hyperplanes and exhibits its ability in improving the performance of hyperbolic SVM.

Related papers

• Fully Hyperbolic Rotation for Knowledge Graph Embedding [12.69417276887153]
  We propose a novel fully hyperbolic model designed for knowledge graph embedding. Our model considers each relation in knowledge graphs as a Lorentz rotation from the head entity to the tail entity. Our model achieves competitive results with fewer parameters.
  arXiv  Detail & Related papers  (2024-11-06T02:41:26Z)
• Enhance Hyperbolic Representation Learning via Second-order Pooling [8.798965454017988]
  We introduce second-order pooling into hyperbolic representation learning. It naturally increases the distance between samples without compromising the generalization ability of the input features. We propose a kernel approximation regularization, which enables the low-dimensional bilinear features to approximate the kernel function well in low-dimensional space.
  arXiv  Detail & Related papers  (2024-10-29T13:17:43Z)
• Hyperbolic Fine-tuning for Large Language Models [56.54715487997674]
  This study investigates the non-Euclidean characteristics of large language models (LLMs) We show that token embeddings exhibit a high degree of hyperbolicity, indicating a latent tree-like structure in the embedding space. We introduce a new method called hyperbolic low-rank efficient fine-tuning, HypLoRA, that performs low-rank adaptation directly on the hyperbolic manifold.
  arXiv  Detail & Related papers  (2024-10-05T02:58:25Z)
• Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin [49.12496652756007]
  We show that the best few-shot results are attained for hyperbolic embeddings at a common hyperbolic radius. In contrast to prior benchmark results, we demonstrate that better performance can be achieved by a fixed-radius encoder equipped with the Euclidean metric.
  arXiv  Detail & Related papers  (2023-09-18T14:51:46Z)
• Modeling the space-time correlation of pulsed twin beams [68.8204255655161]
  Entangled twin-beams generated by parametric down-conversion are among the favorite sources for imaging-oriented applications. We propose a semi-analytic model which aims to bridge the gap between time-consuming numerical simulations and the unrealistic plane-wave pump theory.
  arXiv  Detail & Related papers  (2023-01-18T11:29:49Z)
• HRCF: Enhancing Collaborative Filtering via Hyperbolic Geometric Regularization [52.369435664689995]
  We introduce a textitHyperbolic Regularization powered Collaborative Filtering (HRCF) and design a geometric-aware hyperbolic regularizer. Specifically, the proposal boosts optimization procedure via the root alignment and origin-aware penalty. Our proposal is able to tackle the over-smoothing problem caused by hyperbolic aggregation and also brings the models a better discriminative ability.
  arXiv  Detail & Related papers  (2022-04-18T06:11:44Z)
• Fully Hyperbolic Neural Networks [63.22521652077353]
  We propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model. We show that our method has better performance for building both shallow and deep networks.
  arXiv  Detail & Related papers  (2021-05-31T03:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.