The Numerical Stability of Hyperbolic Representation Learning
- URL: http://arxiv.org/abs/2211.00181v3
- Date: Wed, 28 Jun 2023 02:54:30 GMT
- Title: The Numerical Stability of Hyperbolic Representation Learning
- Authors: Gal Mishne, Zhengchao Wan, Yusu Wang, Sheng Yang
- Abstract summary: We analyze the limitations of two popular models for the hyperbolic space, namely, the Poincar'e ball and the Lorentz model.
We extend this Euclidean parametrization to hyperbolic hyperplanes and exhibit its ability to improve the performance of hyperbolic SVM.
- Score: 36.32817250000654
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Given the exponential growth of the volume of the ball w.r.t. its radius, the
hyperbolic space is capable of embedding trees with arbitrarily small
distortion and hence has received wide attention for representing hierarchical
datasets. However, this exponential growth property comes at a price of
numerical instability such that training hyperbolic learning models will
sometimes lead to catastrophic NaN problems, encountering unrepresentable
values in floating point arithmetic. In this work, we carefully analyze the
limitation of two popular models for the hyperbolic space, namely, the
Poincar\'e ball and the Lorentz model. We first show that, under the 64 bit
arithmetic system, the Poincar\'e ball has a relatively larger capacity than
the Lorentz model for correctly representing points. Then, we theoretically
validate the superiority of the Lorentz model over the Poincar\'e ball from the
perspective of optimization. Given the numerical limitations of both models, we
identify one Euclidean parametrization of the hyperbolic space which can
alleviate these limitations. We further extend this Euclidean parametrization
to hyperbolic hyperplanes and exhibits its ability in improving the performance
of hyperbolic SVM.
Related papers
- Hyperbolic vs Euclidean Embeddings in Few-Shot Learning: Two Sides of
the Same Coin [49.12496652756007]
We show that the best few-shot results are attained for hyperbolic embeddings at a common hyperbolic radius.
In contrast to prior benchmark results, we demonstrate that better performance can be achieved by a fixed-radius encoder equipped with the Euclidean metric.
arXiv Detail & Related papers (2023-09-18T14:51:46Z) - Modeling the space-time correlation of pulsed twin beams [68.8204255655161]
Entangled twin-beams generated by parametric down-conversion are among the favorite sources for imaging-oriented applications.
We propose a semi-analytic model which aims to bridge the gap between time-consuming numerical simulations and the unrealistic plane-wave pump theory.
arXiv Detail & Related papers (2023-01-18T11:29:49Z) - High-dimensional limit theorems for SGD: Effective dynamics and critical
scaling [6.950316788263433]
We prove limit theorems for the trajectories of summary statistics of gradient descent (SGD)
We show a critical scaling regime for the step-size, below which the effective ballistic dynamics matches gradient flow for the population loss.
About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate.
arXiv Detail & Related papers (2022-06-08T17:42:18Z) - HRCF: Enhancing Collaborative Filtering via Hyperbolic Geometric
Regularization [52.369435664689995]
We introduce a textitHyperbolic Regularization powered Collaborative Filtering (HRCF) and design a geometric-aware hyperbolic regularizer.
Specifically, the proposal boosts optimization procedure via the root alignment and origin-aware penalty.
Our proposal is able to tackle the over-smoothing problem caused by hyperbolic aggregation and also brings the models a better discriminative ability.
arXiv Detail & Related papers (2022-04-18T06:11:44Z) - Nested Hyperbolic Spaces for Dimensionality Reduction and Hyperbolic NN
Design [8.250374560598493]
Hyperbolic neural networks have been popular in the recent past due to their ability to represent hierarchical data sets effectively and efficiently.
The challenge in developing these networks lies in the nonlinearity of the embedding space namely, the Hyperbolic space.
We present a novel fully hyperbolic neural network which uses the concept of projections (embeddings) followed by an intrinsic aggregation and a nonlinearity all within the hyperbolic space.
arXiv Detail & Related papers (2021-12-03T03:20:27Z) - Fully Hyperbolic Neural Networks [63.22521652077353]
We propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model.
We show that our method has better performance for building both shallow and deep networks.
arXiv Detail & Related papers (2021-05-31T03:36:49Z) - Hyperbolic Variational Graph Neural Network for Modeling Dynamic Graphs [77.33781731432163]
We learn dynamic graph representation in hyperbolic space, for the first time, which aims to infer node representations.
We present a novel Hyperbolic Variational Graph Network, referred to as HVGNN.
In particular, to model the dynamics, we introduce a Temporal GNN (TGNN) based on a theoretically grounded time encoding approach.
arXiv Detail & Related papers (2021-04-06T01:44:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.