Geometric Hyena Networks for Large-scale Equivariant Learning

• URL: http://arxiv.org/abs/2505.22560v1
• Date: Wed, 28 May 2025 16:38:35 GMT
• Title: Geometric Hyena Networks for Large-scale Equivariant Learning
• Authors: Artem Moskalev, Mangal Prakash, Junjie Xu, Tianyu Cui, Rui Liao, Tommaso Mansi,
• Abstract summary: We introduce Geometric Hyena, the first equivariant long-convolutional model for geometric systems.<n>Our model processes the geometric context of 30k tokens 20x faster than the equivariant transformer.
• Score: 7.878315628263448
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Processing global geometric context while preserving equivariance is crucial when modeling biological, chemical, and physical systems. Yet, this is challenging due to the computational demands of equivariance and global context at scale. Standard methods such as equivariant self-attention suffer from quadratic complexity, while local methods such as distance-based message passing sacrifice global information. Inspired by the recent success of state-space and long-convolutional models, we introduce Geometric Hyena, the first equivariant long-convolutional model for geometric systems. Geometric Hyena captures global geometric context at sub-quadratic complexity while maintaining equivariance to rotations and translations. Evaluated on all-atom property prediction of large RNA molecules and full protein molecular dynamics, Geometric Hyena outperforms existing equivariant models while requiring significantly less memory and compute that equivariant self-attention. Notably, our model processes the geometric context of 30k tokens 20x faster than the equivariant transformer and allows 72x longer context within the same budget.

Related papers

• Geometry-Informed Neural Operator Transformer [0.8906214436849201]
  This work introduces the Geometry-Informed Neural Operator Transformer (GINOT), which integrates the transformer architecture with the neural operator framework to enable forward predictions on arbitrary geometries.<n>The performance of GINOT is validated on multiple challenging datasets, showcasing its high accuracy and strong generalization capabilities for complex and arbitrary 2D and 3D geometries.
  arXiv  Detail & Related papers  (2025-04-28T03:39:27Z)
• SE(3)-Hyena Operator for Scalable Equivariant Learning [5.354533854744212]
  We introduce SE(3)-Hyena, an equivariant long-convolutional model based on the Hyena operator. Our model processes the geometric context of 20k tokens x3.5 times faster than the equivariant transformer.
  arXiv  Detail & Related papers  (2024-07-01T07:56:48Z)
• Approximately Equivariant Neural Processes [47.14384085714576]
  When modelling real-world data, learning problems are often not exactly equivariant, but only approximately. Current approaches to achieving this cannot usually be applied out-of-the-box to any architecture and symmetry group. We develop a general approach to achieving this using existing equivariant architectures.
  arXiv  Detail & Related papers  (2024-06-19T12:17:14Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• Revisiting Transformation Invariant Geometric Deep Learning: Are Initial Representations All You Need? [80.86819657126041]
  We show that transformation-invariant and distance-preserving initial representations are sufficient to achieve transformation invariance. Specifically, we realize transformation-invariant and distance-preserving initial point representations by modifying multi-dimensional scaling. We prove that TinvNN can strictly guarantee transformation invariance, being general and flexible enough to be combined with the existing neural networks.
  arXiv  Detail & Related papers  (2021-12-23T03:52:33Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• GeoMol: Torsional Geometric Generation of Molecular 3D Conformer Ensembles [60.12186997181117]
  Prediction of a molecule's 3D conformer ensemble from the molecular graph holds a key role in areas of cheminformatics and drug discovery. Existing generative models have several drawbacks including lack of modeling important molecular geometry elements. We propose GeoMol, an end-to-end, non-autoregressive and SE(3)-invariant machine learning approach to generate 3D conformers.
  arXiv  Detail & Related papers  (2021-06-08T14:17:59Z)
• Generalizing Convolutional Neural Networks for Equivariance to Lie Groups on Arbitrary Continuous Data [52.78581260260455]
  We propose a general method to construct a convolutional layer that is equivariant to transformations from any specified Lie group. We apply the same model architecture to images, ball-and-stick molecular data, and Hamiltonian dynamical systems.
  arXiv  Detail & Related papers  (2020-02-25T17:40:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.