EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

• URL: http://arxiv.org/abs/2604.09130v1
• Date: Fri, 10 Apr 2026 09:12:16 GMT
• Title: EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers
• Abstract summary: We introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer.<n>We optimize the software implementation, achieving $1.75times$ speedup.<n>We propose SwiGLU-$S2$ activations to incorporate many-body interactions for better theoretical expressivity.
• Score: 70.68282700439444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As $SE(3)$-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving $1.75\times$ speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-$S^2$ activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling $S^2$ grids. Together, SwiGLU-$S^2$ activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.

Related papers

• Efficient Hybrid SE(3)-Equivariant Visuomotor Flow Policy via Spherical Harmonics for Robot Manipulation [51.48086451600577]
  E3Flow is a novel framework that addresses the limitations of equivariant diffusion policies.<n>It unifies efficient rectified flow with stable, multi-modal equivariant learning for the first time.<n>E3Flow achieves a 3.12% improvement in average success rate over the state-of-the-art Spherical Diffusion Policy.
  arXiv  Detail & Related papers  (2026-03-24T14:00:36Z)
• E2Former-V2: On-the-Fly Equivariant Attention with Linear Activation Memory [13.451231889715542]
  Equivariant Graph Neural Networks (EGNNs) have become a widely used approach for modeling 3D atomistic systems.<n>We introduce textbfE2Former-V2, a scalable architecture that integrates algebraic sparsity with hardware-aware execution.<n>E2Former-V2 maintains comparable predictive performance while notably accelerating inference.
  arXiv  Detail & Related papers  (2026-01-23T10:20:08Z)
• Speedy Deformable 3D Gaussian Splatting: Fast Rendering and Compression of Dynamic Scenes [57.69608119350651]
  Recent extensions of 3D Gaussian Splatting (3DGS) to dynamic scenes achieve high-quality novel view synthesis by using neural networks to predict the time-varying deformation of each Gaussian.<n>However, performing per-Gaussian neural inference at every frame poses a significant bottleneck, limiting rendering speed and increasing memory and compute requirements.<n>We present Speedy Deformable 3D Gaussian Splatting (SpeeDe3DGS), a general pipeline for accelerating the rendering speed of dynamic 3DGS and 4DGS representations by reducing neural inference through two complementary techniques.
  arXiv  Detail & Related papers  (2025-06-09T16:30:48Z)
• E2Former: An Efficient and Equivariant Transformer with Linear-Scaling Tensor Products [30.856584261032207]
  We introduce E2Former, an equivariant and efficient transformer architecture that incorporates the Wigner $6j$ convolution (Wigner $6j$ Conv)<n>By shifting the computational burden from edges to nodes, the Wigner $6j$ Conv reduces the complexity from $O(|mathcalE|)$ to $ O(| mathcalV|)$ while preserving both the model's expressive power and rotational equivariance.<n>This development could suggest a promising direction for scalable and efficient molecular modeling.
  arXiv  Detail & Related papers  (2025-01-31T15:22:58Z)
• Geometric Algebra Planes: Convex Implicit Neural Volumes [70.12234371845445]
  We show that GA-Planes is equivalent to a sparse low-rank factor plus low-resolution matrix. We also show that GA-Planes can be adapted for many existing representations.
  arXiv  Detail & Related papers  (2024-11-20T18:21:58Z)
• Kolmogorov-Arnold Transformer [72.88137795439407]
  We introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces layers with Kolmogorov-Arnold Network (KAN) layers. We identify three key challenges: (C1) Base function, (C2) Inefficiency, and (C3) Weight. With these designs, KAT outperforms traditional-based transformers.
  arXiv  Detail & Related papers  (2024-09-16T17:54:51Z)
• EquiformerV2: Improved Equivariant Transformer for Scaling to Higher-Degree Representations [9.718771797861908]
  We propose EquiformerV2, which outperforms previous state-of-the-art methods on large-scale OC20 dataset by up to $9%$ on forces. We also compare EquiformerV2 with Equiformer on QM9 and OC20 S2EF-2M datasets to better understand the performance gain brought by higher degrees.
  arXiv  Detail & Related papers  (2023-06-21T07:01:38Z)
• Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs [3.1618838742094457]
  equivariant convolutions increase significantly in computational complexity as higher-order tensors are used. We propose a graph neural network utilizing our novel approach to equivariant convolutions, which achieves state-of-the-art results on the large-scale OC-20 and OC-22 datasets.
  arXiv  Detail & Related papers  (2023-02-07T18:16:13Z)
• Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphs [3.0603554929274908]
  3D-related inductive biases are indispensable to graph neural networks operating on 3D atomistic graphs such as molecules. Inspired by the success of Transformers in various domains, we study how to incorporate these inductive biases into Transformers. We present Equiformer, a graph neural network leveraging the strength of Transformer architectures.
  arXiv  Detail & Related papers  (2022-06-23T21:40:37Z)
• SE(3)-Transformers: 3D Roto-Translation Equivariant Attention Networks [71.55002934935473]
  We introduce the SE(3)-Transformer, a variant of the self-attention module for 3D point clouds and graphs, which is equivariant under continuous 3D roto-translations. We evaluate our model on a toy N-body particle simulation dataset, showcasing the robustness of the predictions under rotations of the input.
  arXiv  Detail & Related papers  (2020-06-18T13:23:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.