Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries

• URL: http://arxiv.org/abs/2504.08896v1
• Date: Fri, 11 Apr 2025 18:07:33 GMT
• Title: Position: Beyond Euclidean -- Foundation Models Should Embrace Non-Euclidean Geometries
• Authors: Neil He, Jiahong Liu, Buze Zhang, Ngoc Bui, Ali Maatouk, Menglin Yang, Irwin King, Melanie Weber, Rex Ying,
• Abstract summary: Euclidean space has been the de facto geometric setting for machine learning architectures.<n>At a large scale, real-world data often exhibit inherently non-Euclidean structures, such as multi-way relationships, hierarchies, symmetries, and non-isotropic scaling.<n>This paper argues that moving beyond Euclidean geometry is not merely an optional enhancement but a necessity to maintain the scaling law for the next-generation of foundation models.
• Score: 42.83280708842304
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In the era of foundation models and Large Language Models (LLMs), Euclidean space has been the de facto geometric setting for machine learning architectures. However, recent literature has demonstrated that this choice comes with fundamental limitations. At a large scale, real-world data often exhibit inherently non-Euclidean structures, such as multi-way relationships, hierarchies, symmetries, and non-isotropic scaling, in a variety of domains, such as languages, vision, and the natural sciences. It is challenging to effectively capture these structures within the constraints of Euclidean spaces. This position paper argues that moving beyond Euclidean geometry is not merely an optional enhancement but a necessity to maintain the scaling law for the next-generation of foundation models. By adopting these geometries, foundation models could more efficiently leverage the aforementioned structures. Task-aware adaptability that dynamically reconfigures embeddings to match the geometry of downstream applications could further enhance efficiency and expressivity. Our position is supported by a series of theoretical and empirical investigations of prevalent foundation models.Finally, we outline a roadmap for integrating non-Euclidean geometries into foundation models, including strategies for building geometric foundation models via fine-tuning, training from scratch, and hybrid approaches.

Related papers

• Geometry Distributions [51.4061133324376]
  We propose a novel geometric data representation that models geometry as distributions. Our approach uses diffusion models with a novel network architecture to learn surface point distributions. We evaluate our representation qualitatively and quantitatively across various object types, demonstrating its effectiveness in achieving high geometric fidelity.
  arXiv  Detail & Related papers  (2024-11-25T04:06:48Z)
• Bridging Geometric States via Geometric Diffusion Bridge [79.60212414973002]
  We introduce the Geometric Diffusion Bridge (GDB), a novel generative modeling framework that accurately bridges initial and target geometric states. GDB employs an equivariant diffusion bridge derived by a modified version of Doob's $h$-transform for connecting geometric states. We show that GDB surpasses existing state-of-the-art approaches, opening up a new pathway for accurately bridging geometric states.
  arXiv  Detail & Related papers  (2024-10-31T17:59:53Z)
• Score-based pullback Riemannian geometry [10.649159213723106]
  We propose a framework for data-driven Riemannian geometry that is scalable in both geometry and learning. We produce high-quality geodesics through the data support and reliably estimates the intrinsic dimension of the data manifold. Our framework can naturally be used with anisotropic normalizing flows by adopting isometry regularization during training.
  arXiv  Detail & Related papers  (2024-10-02T18:52:12Z)
• Beyond Euclid: An Illustrated Guide to Modern Machine Learning with Geometric, Topological, and Algebraic Structures [3.5357133304100326]
  Modern machine learning encounters richly structured data that is inherently nonEuclidean. Extracting knowledge from such non-Euclidean data requires a broader mathematical perspective. We propose a graphical taxonomy that integrates recent advances into an intuitive unified framework.
  arXiv  Detail & Related papers  (2024-07-12T17:48:36Z)
• Deep Learning of Multivariate Extremes via a Geometric Representation [0.0]
  We study the study of geometric extremes, where extremal dependence properties are inferred from the limiting shapes of scaled sample clouds. We propose a range of novel theoretical results to aid with the implementation of the geometric extremes framework. We introduce the first approach to modelling limit sets using deep learning neural networks.
  arXiv  Detail & Related papers  (2024-06-28T14:05:47Z)
• Generalizing Knowledge Graph Embedding with Universal Orthogonal Parameterization [22.86465452511445]
  GoldE is a powerful framework for knowledge graph embedding. It features a universal orthogonal parameterization based on a generalized form of Householder reflection. It achieves state-of-the-art performance on three standard benchmarks.
  arXiv  Detail & Related papers  (2024-05-14T12:26:19Z)
• Adaptive Surface Normal Constraint for Geometric Estimation from Monocular Images [56.86175251327466]
  We introduce a novel approach to learn geometries such as depth and surface normal from images while incorporating geometric context. Our approach extracts geometric context that encodes the geometric variations present in the input image and correlates depth estimation with geometric constraints. Our method unifies depth and surface normal estimations within a cohesive framework, which enables the generation of high-quality 3D geometry from images.
  arXiv  Detail & Related papers  (2024-02-08T17:57:59Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• Exploring Data Geometry for Continual Learning [64.4358878435983]
  We study continual learning from a novel perspective by exploring data geometry for the non-stationary stream of data. Our method dynamically expands the geometry of the underlying space to match growing geometric structures induced by new data. Experiments show that our method achieves better performance than baseline methods designed in Euclidean space.
  arXiv  Detail & Related papers  (2023-04-08T06:35:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.