The Numerical Stability of Hyperbolic Representation Learning

• URL: http://arxiv.org/abs/2211.00181v3
• Date: Wed, 28 Jun 2023 02:54:30 GMT
• Title: The Numerical Stability of Hyperbolic Representation Learning
• Authors: Gal Mishne, Zhengchao Wan, Yusu Wang, Sheng Yang
• Abstract summary: We analyze the limitations of two popular models for the hyperbolic space, namely, the Poincar'e ball and the Lorentz model. We extend this Euclidean parametrization to hyperbolic hyperplanes and exhibit its ability to improve the performance of hyperbolic SVM.
• Score: 36.32817250000654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given the exponential growth of the volume of the ball w.r.t. its radius, the hyperbolic space is capable of embedding trees with arbitrarily small distortion and hence has received wide attention for representing hierarchical datasets. However, this exponential growth property comes at a price of numerical instability such that training hyperbolic learning models will sometimes lead to catastrophic NaN problems, encountering unrepresentable values in floating point arithmetic. In this work, we carefully analyze the limitation of two popular models for the hyperbolic space, namely, the Poincar\'e ball and the Lorentz model. We first show that, under the 64 bit arithmetic system, the Poincar\'e ball has a relatively larger capacity than the Lorentz model for correctly representing points. Then, we theoretically validate the superiority of the Lorentz model over the Poincar\'e ball from the perspective of optimization. Given the numerical limitations of both models, we identify one Euclidean parametrization of the hyperbolic space which can alleviate these limitations. We further extend this Euclidean parametrization to hyperbolic hyperplanes and exhibits its ability in improving the performance of hyperbolic SVM.

Related papers

• Intrinsic Lorentz Neural Network [49.83037691286893]
  Real-world data frequently exhibit latent hierarchical structures, which can be naturally represented by hyperbolic geometry.<n>We propose the emphIntrinsic Lorentz Neural Network (ILNN), a fully intrinsic hyperbolic architecture that conducts all computations within the Lorentz model.<n>At its core, the network introduces a novel emphpoint-to-hyperplane fully connected layer (FC), replacing traditional Euclidean affine logits with closed-form hyperbolic distances.
  arXiv  Detail & Related papers  (2026-02-27T12:48:05Z)
• Fast and Geometrically Grounded Lorentz Neural Networks [44.564864487582525]
  We develop a formulation of hyperbolic neural networks that is both efficient and captures the key properties of hyperbolic space.<n>We prove that, with the current formulation of Lorentz linear layers, the hyperbolic norms of the outputs scale logarithmically with the number of gradient descent steps.<n>Our new formulation, together with further efficiencies through Lorentzian activation functions and a new caching strategy results in neural networks fully abiding by hyperbolic geometry.
  arXiv  Detail & Related papers  (2026-01-29T10:44:32Z)
• Hyperbolic Coarse-to-Fine Few-Shot Class-Incremental Learning [6.461974916262412]
  This work focuses on the Coarse-To-Fine Few-Shot Class-Incremental Learning (C2FSCIL) task.<n>To better interpret the "coarse-to-fine" paradigm, we propose embedding the feature extractor into hyperbolic space.<n> Experiments on C2FSCIL benchmarks show that our method effectively improves both coarse and fine class accuracies.
  arXiv  Detail & Related papers  (2025-09-23T01:12:21Z)
• Multi-Hop Reasoning for Question Answering with Hyperbolic Representations [7.312170216336085]
  We compare the capacity of hyperbolic space versus Euclidean space in multi-hop reasoning.<n>Our results show that the former consistently outperforms the latter across a diverse set of datasets.<n>Our findings suggest that hyperbolic representations can be significantly more advantageous when the datasets exhibit a more hierarchical structure.
  arXiv  Detail & Related papers  (2025-07-04T14:39:01Z)
• Fully Hyperbolic Rotation for Knowledge Graph Embedding [12.69417276887153]
  We propose a novel fully hyperbolic model designed for knowledge graph embedding. Our model considers each relation in knowledge graphs as a Lorentz rotation from the head entity to the tail entity. Our model achieves competitive results with fewer parameters.
  arXiv  Detail & Related papers  (2024-11-06T02:41:26Z)
• Enhance Hyperbolic Representation Learning via Second-order Pooling [8.798965454017988]
  We introduce second-order pooling into hyperbolic representation learning. It naturally increases the distance between samples without compromising the generalization ability of the input features. We propose a kernel approximation regularization, which enables the low-dimensional bilinear features to approximate the kernel function well in low-dimensional space.
  arXiv  Detail & Related papers  (2024-10-29T13:17:43Z)
• Hyperbolic Fine-tuning for Large Language Models [56.54715487997674]
  This study investigates the non-Euclidean characteristics of large language models (LLMs) We show that token embeddings exhibit a high degree of hyperbolicity, indicating a latent tree-like structure in the embedding space. We introduce a new method called hyperbolic low-rank efficient fine-tuning, HypLoRA, that performs low-rank adaptation directly on the hyperbolic manifold.
  arXiv  Detail & Related papers  (2024-10-05T02:58:25Z)
• Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning [70.0681902472251]
  We propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees.
  arXiv  Detail & Related papers  (2023-10-27T15:31:42Z)
• Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of the Same Coin [49.12496652756007]
  We show that the best few-shot results are attained for hyperbolic embeddings at a common hyperbolic radius. In contrast to prior benchmark results, we demonstrate that better performance can be achieved by a fixed-radius encoder equipped with the Euclidean metric.
  arXiv  Detail & Related papers  (2023-09-18T14:51:46Z)
• Modeling the space-time correlation of pulsed twin beams [68.8204255655161]
  Entangled twin-beams generated by parametric down-conversion are among the favorite sources for imaging-oriented applications. We propose a semi-analytic model which aims to bridge the gap between time-consuming numerical simulations and the unrealistic plane-wave pump theory.
  arXiv  Detail & Related papers  (2023-01-18T11:29:49Z)
• HRCF: Enhancing Collaborative Filtering via Hyperbolic Geometric Regularization [52.369435664689995]
  We introduce a textitHyperbolic Regularization powered Collaborative Filtering (HRCF) and design a geometric-aware hyperbolic regularizer. Specifically, the proposal boosts optimization procedure via the root alignment and origin-aware penalty. Our proposal is able to tackle the over-smoothing problem caused by hyperbolic aggregation and also brings the models a better discriminative ability.
  arXiv  Detail & Related papers  (2022-04-18T06:11:44Z)
• Fully Hyperbolic Neural Networks [63.22521652077353]
  We propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model. We show that our method has better performance for building both shallow and deep networks.
  arXiv  Detail & Related papers  (2021-05-31T03:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.