Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform

• URL: http://arxiv.org/abs/2607.21271v2
• Date: Fri, 24 Jul 2026 07:27:15 GMT
• Title: Flash EQ-Linear: Accelerating Equivariant Linear Layers via Group-wise Discrete Fourier Transform
• Abstract summary: Equivariant networks embed geometric symmetries as structural priors through weight sharing.<n>Existing implementations unroll the structured weights into dense matrices and dispatch them to generic dense kernels.<n>This is the first time equivariant networks strictly dominate their non-equivariant counterparts along all three axes simultaneously.
• Score: 65.89510337046077
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Equivariant networks embed geometric symmetries as structural priors through weight sharing, achieving remarkable parameter efficiency across vision tasks. However, this parameter efficiency does not translate into compute efficiency: existing implementations unroll the structured weights into dense matrices and dispatch them to generic dense kernels, so the FLOPs of an equivariant layer are no smaller than those of a non-equivariant counterpart. In this paper, we observe that the equivariant linear (EQ-Linear) layer---the most fundamental and frequently used module in modern equivariant architectures---is essentially a circular convolution along the group dimension composed with a linear transform along the channel dimension. Building on this observation, we propose Flash EQ-Linear, an exact acceleration algorithm that reduces the complexity from $\mathcal{O}(NDC)$ to $\mathcal{O}(NDC/T)$ by combining the Fourier convolution theorem along the group dimension with the conjugate symmetry of the real DFT. We further provide dedicated CUDA kernels for Flash EQ-Linear, covering both forward and backward passes and both FP32 and FP16 precision. At the operator level, Flash EQ-Linear achieves up to ${2\times}$ forward speedup over PyTorch's F.linear; at the network level, Flash EQ-ViT and Flash EQ-Swin achieve up to ${1.7\times}$ end-to-end speedup over both equivariant and non-equivariant baselines. To our knowledge, this is the first time equivariant networks strictly dominate their non-equivariant counterparts along all three axes simultaneously: accuracy, parameter efficiency, and inference speed.Code is available at https://github.com/zhongchenzhao/FlashEQLinear.

Related papers

• MeshFlow: Mesh Generation with Equivariant Flow Matching [99.07593100078935]
  MeshFlow learns to generate triangle meshes directly as triangle soups.<n>We adopt equivariant optimal-transport flow matching models that respect the key symmetries of triangle soups.<n>MeshFlow achieves mesh quality comparable to state-of-the-art autoregressive mesh generators.
  arXiv  Detail & Related papers  (2026-06-22T15:35:30Z)
• Fourier Neural Operators with rank-1 lattice points and hyperbolic cross [40.78425934640374]
  The emphFourier neural operator (FNO) is a neural network architecture that learns mappings between function spaces.<n>We prove that the generalization error of the FNO can be improved by replacing spatial tensor product grids with rank-1 lattice points.<n>We achieve more accurate and efficient approximations from fewer network parameters, fewer spatial points, and fewer training samples.
  arXiv  Detail & Related papers  (2026-06-07T22:55:39Z)
• Fourier Feature Pyramids for Physics-Informed Neural Networks [47.75068288153803]
  We present an improved neural field architecture for solving partial differential equations (PDEs)<n>We call our model beignet (Bandlimited Embedding with Interpolated Grid Network)<n>We demonstrate that beignet finds significantly more accurate solutions on PDE benchmarks using fewer parameters than state-of-the-art PINN methods.
  arXiv  Detail & Related papers  (2026-05-22T23:12:56Z)
• Scale-Invariant Neural Network Optimization: Norm Geometry and Heavy-Tailed Noise [12.977441534320041]
  We show that a scale-invariant first-order method with spectral norm requires $(minm, n-frac3p-2p-1) calls.<n>We prove that a batched method achieves the matching bound of $(minm, n-frac5p2p-2p-2)$ when the norm is spectral and the Hessian is Lipschitz.
  arXiv  Detail & Related papers  (2026-05-18T15:13:18Z)
• Quantum-Inspired Algorithms beyond Unitary Circuits: the Laplace Transform [0.0]
  Quantum-inspired algorithms can deliver substantial speedups over classical state-of-the-art methods.<n>We introduce a tensor-network approach to compute the discrete Laplace transform, a non-unitary, aperiodic transform.<n>We demonstrate simulations up to $N=230$ input data points, with up to $260$ output data points, and quantify how bond dimension controls runtime and accuracy.
  arXiv  Detail & Related papers  (2026-01-25T07:19:56Z)
• Flopping for FLOPs: Leveraging equivariance for computational efficiency [13.909111585519652]
  This paper introduces new equivariant neural networks that preserve symmetry while maintaining a comparable number of floating-point operations (FLOPs) per parameter to standard non-equivariant networks.<n>Our approach reduces both FLOPs and wall-clock time, providing a practical solution for efficient, scalable symmetry-aware architectures.
  arXiv  Detail & Related papers  (2025-02-07T18:53:03Z)
• Transformers as Support Vector Machines [54.642793677472724]
  We establish a formal equivalence between the optimization geometry of self-attention and a hard-margin SVM problem. We characterize the implicit bias of 1-layer transformers optimized with gradient descent. We believe these findings inspire the interpretation of transformers as a hierarchy of SVMs that separates and selects optimal tokens.
  arXiv  Detail & Related papers  (2023-08-31T17:57:50Z)
• Adaptive Fourier Neural Operators: Efficient Token Mixers for Transformers [55.90468016961356]
  We propose an efficient token mixer that learns to mix in the Fourier domain. AFNO is based on a principled foundation of operator learning. It can handle a sequence size of 65k and outperforms other efficient self-attention mechanisms.
  arXiv  Detail & Related papers  (2021-11-24T05:44:31Z)
• Frame Averaging for Invariant and Equivariant Network Design [50.87023773850824]
  We introduce Frame Averaging (FA), a framework for adapting known (backbone) architectures to become invariant or equivariant to new symmetry types. We show that FA-based models have maximal expressive power in a broad setting. We propose a new class of universal Graph Neural Networks (GNNs), universal Euclidean motion invariant point cloud networks, and Euclidean motion invariant Message Passing (MP) GNNs.
  arXiv  Detail & Related papers  (2021-10-07T11:05:23Z)
• Nonlinearities in Steerable SO(2)-Equivariant CNNs [7.552100672006172]
  We apply harmonic distortion analysis to illuminate the effect of nonlinearities on representations of SO(2). We develop a novel FFT-based algorithm for computing representations of non-linearly transformed activations. In experiments with 2D and 3D data, we obtain results that compare favorably to the state-of-the-art in terms of accuracy while continuous symmetry and exact equivariance.
  arXiv  Detail & Related papers  (2021-09-14T17:53:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.