Lorentzian Residual Neural Networks

• URL: http://arxiv.org/abs/2412.14695v2
• Date: Sun, 12 Jan 2025 05:47:25 GMT
• Title: Lorentzian Residual Neural Networks
• Authors: Neil He, Menglin Yang, Rex Ying,
• Abstract summary: We introduce LResNet, a novel Lorentzian residual neural network based on the weighted Lorentzian centroid in the Lorentz model of hyperbolic geometry.<n>Our method enables the efficient integration of residual connections in hyperbolic neural networks while preserving their hierarchical representation capabilities.<n>Our findings highlight the potential of LResNet for building more expressive neural networks in hyperbolic embedding space.
• Score: 15.257990326035694
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Hyperbolic neural networks have emerged as a powerful tool for modeling hierarchical data structures prevalent in real-world datasets. Notably, residual connections, which facilitate the direct flow of information across layers, have been instrumental in the success of deep neural networks. However, current methods for constructing hyperbolic residual networks suffer from limitations such as increased model complexity, numerical instability, and errors due to multiple mappings to and from the tangent space. To address these limitations, we introduce LResNet, a novel Lorentzian residual neural network based on the weighted Lorentzian centroid in the Lorentz model of hyperbolic geometry. Our method enables the efficient integration of residual connections in Lorentz hyperbolic neural networks while preserving their hierarchical representation capabilities. We demonstrate that our method can theoretically derive previous methods while offering improved stability, efficiency, and effectiveness. Extensive experiments on both graph and vision tasks showcase the superior performance and robustness of our method compared to state-of-the-art Euclidean and hyperbolic alternatives. Our findings highlight the potential of LResNet for building more expressive neural networks in hyperbolic embedding space as a generally applicable method to multiple architectures, including CNNs, GNNs, and graph Transformers.

Related papers

• sHGCN: Simplified hyperbolic graph convolutional neural networks [0.0]
  Hyperbolic geometry has emerged as a powerful tool for modeling complex, structured data.<n>We show that streamlined hyperbolic operations can lead to substantial gains in computational speed and predictive accuracy.
  arXiv  Detail & Related papers  (2025-06-17T11:58:07Z)
• Training Neural Networks by Optimizing Neuron Positions [39.682133213072554]
  We propose a parameter-efficient neural architecture where neurons are embedded in Euclidean space.<n>During training, their positions are optimized and synaptic weights are determined as the inverse of the spatial distance between connected neurons.<n>These distance-dependent wiring rules replace traditional learnable weight matrices and significantly reduce the number of parameters while introducing a biologically inspired inductive bias.
  arXiv  Detail & Related papers  (2025-06-16T12:26:13Z)
• Lattice-Based Pruning in Recurrent Neural Networks via Poset Modeling [0.0]
  Recurrent neural networks (RNNs) are central to sequence modeling tasks, yet their high computational complexity poses challenges for scalability and real-time deployment. We introduce a novel framework that models RNNs as partially ordered sets (posets) and constructs corresponding dependency lattices. By identifying meet irreducible neurons, our lattice-based pruning algorithm selectively retains critical connections while eliminating redundant ones.
  arXiv  Detail & Related papers  (2025-02-23T10:11:38Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• Efficient and Flexible Neural Network Training through Layer-wise Feedback Propagation [49.44309457870649]
  Layer-wise Feedback feedback (LFP) is a novel training principle for neural network-like predictors.<n>LFP decomposes a reward to individual neurons based on their respective contributions.<n>Our method then implements a greedy reinforcing approach helpful parts of the network and weakening harmful ones.
  arXiv  Detail & Related papers  (2023-08-23T10:48:28Z)
• Variational Tensor Neural Networks for Deep Learning [0.0]
  We propose an integration of tensor networks (TN) into deep neural networks (NNs) This in turn, results in a scalable tensor neural network (TNN) architecture capable of efficient training over a large parameter space. We validate the accuracy and efficiency of our method by designing TNN models and providing benchmark results for linear and non-linear regressions, data classification and image recognition on MNIST handwritten digits.
  arXiv  Detail & Related papers  (2022-11-26T20:24:36Z)
• Quiver neural networks [5.076419064097734]
  We develop a uniform theoretical approach towards the analysis of various neural network connectivity architectures. Inspired by quiver representation theory in mathematics, this approach gives a compact way to capture elaborate data flows.
  arXiv  Detail & Related papers  (2022-07-26T09:42:45Z)
• Creating Powerful and Interpretable Models withRegression Networks [2.2049183478692584]
  We propose a novel architecture, Regression Networks, which combines the power of neural networks with the understandability of regression analysis. We demonstrate that the models exceed the state-of-the-art performance of interpretable models on several benchmark datasets.
  arXiv  Detail & Related papers  (2021-07-30T03:37:00Z)
• Fully Hyperbolic Neural Networks [63.22521652077353]
  We propose a fully hyperbolic framework to build hyperbolic networks based on the Lorentz model. We show that our method has better performance for building both shallow and deep networks.
  arXiv  Detail & Related papers  (2021-05-31T03:36:49Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)
• Hyperbolic Neural Networks++ [66.16106727715061]
  We generalize the fundamental components of neural networks in a single hyperbolic geometry model, namely, the Poincar'e ball model. Experiments show the superior parameter efficiency of our methods compared to conventional hyperbolic components, and stability and outperformance over their Euclidean counterparts.
  arXiv  Detail & Related papers  (2020-06-15T08:23:20Z)
• Binarized Graph Neural Network [65.20589262811677]
  We develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters. Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches. Experiments indicate that the proposed binarized graph neural network, namely BGN, is orders of magnitude more efficient in terms of both time and space.
  arXiv  Detail & Related papers  (2020-04-19T09:43:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.