E2Former: A Linear-time Efficient and Equivariant Transformer for Scalable Molecular Modeling

• URL: http://arxiv.org/abs/2501.19216v2
• Date: Mon, 03 Feb 2025 18:46:30 GMT
• Title: E2Former: A Linear-time Efficient and Equivariant Transformer for Scalable Molecular Modeling
• Authors: Yunyang Li, Lin Huang, Zhihao Ding, Chu Wang, Xinran Wei, Han Yang, Zun Wang, Chang Liu, Yu Shi, Peiran Jin, Jia Zhang, Mark Gerstein, Tao Qin,
• Abstract summary: We introduce E2Former, an equivariant and efficient transformer architecture that incorporates the Wigner $6j$ convolution (Wigner $6j$ Conv)<n>By shifting the computational burden from edges to nodes, the Wigner $6j$ Conv reduces the complexity from $O(|mathcalE|)$ to $ O(| mathcalV|)$ while preserving both the model's expressive power and rotational equivariance.<n>This development could suggest a promising direction for scalable and efficient molecular modeling.
• Score: 44.75336958712181
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Equivariant Graph Neural Networks (EGNNs) have demonstrated significant success in modeling microscale systems, including those in chemistry, biology and materials science. However, EGNNs face substantial computational challenges due to the high cost of constructing edge features via spherical tensor products, making them impractical for large-scale systems. To address this limitation, we introduce E2Former, an equivariant and efficient transformer architecture that incorporates the Wigner $6j$ convolution (Wigner $6j$ Conv). By shifting the computational burden from edges to nodes, the Wigner $6j$ Conv reduces the complexity from $O(|\mathcal{E}|)$ to $ O(| \mathcal{V}|)$ while preserving both the model's expressive power and rotational equivariance. We show that this approach achieves a 7x-30x speedup compared to conventional $\mathrm{SO}(3)$ convolutions. Furthermore, our empirical results demonstrate that the derived E2Former mitigates the computational challenges of existing approaches without compromising the ability to capture detailed geometric information. This development could suggest a promising direction for scalable and efficient molecular modeling.

Related papers

• Geometric Algebra Planes: Convex Implicit Neural Volumes [70.12234371845445]
  We show that GA-Planes is equivalent to a sparse low-rank factor plus low-resolution matrix. We also show that GA-Planes can be adapted for many existing representations.
  arXiv  Detail & Related papers  (2024-11-20T18:21:58Z)
• Scaling Gaussian Processes for Learning Curve Prediction via Latent Kronecker Structure [16.319561844942886]
  We show that our GP model can match the performance of a Transformer on a learning curve prediction task. Our method only requires $mathcalO(n3 + m3)$ time and $mathcalO(n2 + m2)$ space.
  arXiv  Detail & Related papers  (2024-10-11T20:24:33Z)
• Projection by Convolution: Optimal Sample Complexity for Reinforcement Learning in Continuous-Space MDPs [56.237917407785545]
  We consider the problem of learning an $varepsilon$-optimal policy in a general class of continuous-space Markov decision processes (MDPs) having smooth Bellman operators. Key to our solution is a novel projection technique based on ideas from harmonic analysis. Our result bridges the gap between two popular but conflicting perspectives on continuous-space MDPs.
  arXiv  Detail & Related papers  (2024-05-10T09:58:47Z)
• Learning with Norm Constrained, Over-parameterized, Two-layer Neural Networks [54.177130905659155]
  Recent studies show that a reproducing kernel Hilbert space (RKHS) is not a suitable space to model functions by neural networks. In this paper, we study a suitable function space for over- parameterized two-layer neural networks with bounded norms.
  arXiv  Detail & Related papers  (2024-04-29T15:04:07Z)
• Similarity Equivariant Graph Neural Networks for Homogenization of Metamaterials [3.6443770850509423]
  Soft, porous mechanical metamaterials exhibit pattern transformations that may have important applications in soft robotics, sound reduction and biomedicine. We develop a machine learning-based approach that scales favorably to serve as a surrogate model. We show that this network is more accurate and data-efficient than graph neural networks with fewer symmetries.
  arXiv  Detail & Related papers  (2024-04-26T12:30:32Z)
• Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products [14.984349569810275]
  We propose a systematic approach to accelerate the complexity of the tensor products of irreps. We introduce the Gaunt Product, which serves as a new method to construct efficient equivariant operations. Our experiments on the Open Catalyst Project and 3BPA datasets demonstrate both the increased efficiency and improved performance.
  arXiv  Detail & Related papers  (2024-01-18T18:57:10Z)
• EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations [9.718771797861908]
  We propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to $9%$ on forces. We also compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.
  arXiv  Detail & Related papers  (2023-06-21T07:01:38Z)
• Scalable Transformer for PDE Surrogate Modeling [9.438207505148947]
  Transformer has emerged as a promising tool for surrogate modeling of partial differential equations (PDEs) We propose Factorized Transformer (FactFormer), which is based on an axial factorized kernel integral. We showcase that the proposed model is able to simulate 2D Kolmogorov flow on a $256times 256$ grid and 3D smoke buoyancy on a $64times64times64$ grid with good accuracy and efficiency.
  arXiv  Detail & Related papers  (2023-05-27T19:23:00Z)
• FAENet: Frame Averaging Equivariant GNN for Materials Modeling [123.19473575281357]
  We introduce a flexible framework relying on frameaveraging (SFA) to make any model E(3)-equivariant or invariant through data transformations. We prove the validity of our method theoretically and empirically demonstrate its superior accuracy and computational scalability in materials modeling.
  arXiv  Detail & Related papers  (2023-04-28T21:48:31Z)
• Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs [3.1618838742094457]
  equivariant convolutions increase significantly in computational complexity as higher-order tensors are used. We propose a graph neural network utilizing our novel approach to equivariant convolutions, which achieves state-of-the-art results on the large-scale OC-20 and OC-22 datasets.
  arXiv  Detail & Related papers  (2023-02-07T18:16:13Z)
• Equivalence Between SE(3) Equivariant Networks via Steerable Kernels and Group Convolution [90.67482899242093]
  A wide range of techniques have been proposed in recent years for designing neural networks for 3D data that are equivariant under rotation and translation of the input. We provide an in-depth analysis of both methods and their equivalence and relate the two constructions to multiview convolutional networks. We also derive new TFN non-linearities from our equivalence principle and test them on practical benchmark datasets.
  arXiv  Detail & Related papers  (2022-11-29T03:42:11Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• Efficient Generalized Spherical CNNs [7.819876182082904]
  We present a generalized spherical CNN framework that encompasses various existing approaches and allows them to be leveraged alongside each other. We show that these developments allow the construction of more expressive hybrid models that achieve state-of-the-art accuracy and parameter efficiency on spherical benchmark problems.
  arXiv  Detail & Related papers  (2020-10-09T18:00:05Z)
• Scalable Deep Generative Modeling for Sparse Graphs [105.60961114312686]
  Existing deep neural methods require $Omega(n2)$ complexity by building up the adjacency matrix. We develop a novel autoregressive model, named BiGG, that utilizes this sparsity to avoid generating the full adjacency matrix. During training this autoregressive model can be parallelized with $O(log n)$ synchronization stages.
  arXiv  Detail & Related papers  (2020-06-28T04:37:57Z)
• FANOK: Knockoffs in Linear Time [73.5154025911318]
  We describe a series of algorithms that efficiently implement Gaussian model-X knockoffs to control the false discovery rate on large scale feature selection problems. We test our methods on problems with $p$ as large as $500,000$.
  arXiv  Detail & Related papers  (2020-06-15T21:55:34Z)
• Quantum Algorithms for Simulating the Lattice Schwinger Model [63.18141027763459]
  We give scalable, explicit digital quantum algorithms to simulate the lattice Schwinger model in both NISQ and fault-tolerant settings. In lattice units, we find a Schwinger model on $N/2$ physical sites with coupling constant $x-1/2$ and electric field cutoff $x-1/2Lambda$. We estimate observables which we cost in both the NISQ and fault-tolerant settings by assuming a simple target observable---the mean pair density.
  arXiv  Detail & Related papers  (2020-02-25T19:18:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.