Exact Linear Attention

• URL: http://arxiv.org/abs/2605.18848v2
• Date: Wed, 20 May 2026 08:24:51 GMT
• Title: Exact Linear Attention
• Abstract summary: Exact Linear Attention (ELA) achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions.<n>ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention.<n>We extend the linear attention principle to vision models, yielding YOLO-LAT, which attains up to 4.3x GPU inference speedup and 7.9x parameter reduction with competitive detection accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces Exact Linear Attention (ELA), a mechanism that achieves linear computational complexity for Transformer attention by exploiting the exact decomposition property of kernel functions, thereby eliminating approximation error. We identify and address two key limitations of prior linear attention -- gradient explosion and token attention dilution -- by imposing kernel constraints that ensure non-negativity, discriminability, and geometric interpretability. Several kernel functions are proposed, including the Hadamard Exp Kernel, Summation Squared Euclidean Distance Kernel, and Subtraction Squared Euclidean Distance Kernel, each tailored for specific attention behaviors. Beyond the core attention formulation, the paper presents three engineering innovations: (1) a Hyper-Link structure that replaces traditional residual connections to mitigate gradient degradation; (2) a Memory Lobe module based on bidirectional linear attention, which captures "transformation flow" across layers to implement qualitative memory and an implicit reinforcement learning paradigm; and (3) a routing-score-based bias mechanism for Mixture-of-Experts (MoE) to improve interpretability and semantic alignment. Experimental results demonstrate that ELA achieves up to 6x faster decoding speed and 75% reduction in KV cache memory usage compared to full attention, while maintaining comparable or superior training performance. The proposed memory module accelerates convergence and enhances generalization. Furthermore, we extend the linear attention principle to vision models, yielding YOLO-LAT, which attains up to 4.3x GPU inference speedup and 7.9x parameter reduction with competitive detection accuracy. These results underline the broad applicability of exact linear attention for scaling Transformer models to ultra-long sequences and efficient visual tasks.

Related papers

• Accelerating Regularized Attention Kernel Regression for Spectrum Cartography [1.9651461643699661]
  This paper proposes a Learning-based Attention Kernel Regression (LAKER) algorithm for accelerating regularized attention kernel regression in spectrum cartography.<n>The key idea is to learn a data-dependent preconditioner that captures the inverse spectral structure of the attention kernel system.<n>Experiments demonstrate that LAKER significantly reduces condition numbers by up to three orders of magnitude, accelerates convergence by over twenty-fold compared to baselines, and maintains high reconstruction accuracy.
  arXiv  Detail & Related papers  (2026-04-28T02:27:40Z)
• LaplacianFormer:Rethinking Linear Attention with Laplacian Kernel [27.87296519831803]
  quadratic complexity of softmax attention presents a major obstacle for scaling Transformers to high-resolution vision tasks.<n>We propose LaplacianFormer, a Transformer variant that employs a Laplacian kernel as a principled alternative to softmax.<n>Experiments on ImageNet show LaplacianFormer achieves strong performance-efficiency trade-offs while improving attention.
  arXiv  Detail & Related papers  (2026-04-22T09:04:54Z)
• Rethinking Vector Field Learning for Generative Segmentation [50.08025820235397]
  Taming diffusion models for generative segmentation has attracted increasing attention.<n>We revisit diffusion segmentation from the perspective of vector field learning.<n>We propose a principled vector field reshaping strategy that augments the learned vanishing velocity field with a detached distance-aware correction term.<n>This correction introduces both attractive and repulsive interactions, enhancing gradient magnitudes near centroids while preserving the original diffusion training framework.
  arXiv  Detail & Related papers  (2026-03-19T17:58:19Z)
• Influence Malleability in Linearized Attention: Dual Implications of Non-Convergent NTK Dynamics [0.0]
  We show that linearized attention does not converge to its infinite-width NTK limit.<n>We also show that attention's power and vulnerability share a common origin in its departure from the kernel regime.
  arXiv  Detail & Related papers  (2026-03-13T15:33:34Z)
• Sparse Attention as Compact Kernel Regression [20.026224027434974]
  kernel-theoretic understanding of sparse attention mechanisms is currently missing.<n>We establish a formal correspondence between sparse attention and compact (bounded support) kernels.<n> Experiments with a kernel-regression-based variant of transformers -- Memory Mosaics -- show that kernel-based sparse attention achieves competitive performance.
  arXiv  Detail & Related papers  (2026-01-30T09:45:35Z)
• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• Neuronal Attention Circuit (NAC) for Representation Learning [2.0573301822495553]
  We introduce a CT-Attention mechanism that reformulates attention logits computation as the solution to a linear first-order ODE.<n>We implement NAC in diverse domains, including irregular time-series classification, lane-keeping for autonomous vehicles, and industrial prognostics.
  arXiv  Detail & Related papers  (2025-12-11T04:49:44Z)
• A Representer Theorem for Hawkes Processes via Penalized Least Squares Minimization [31.876688992403647]
  The representer theorem is a cornerstone of kernel methods, which aim to estimate latent functions in reproducing kernel Hilbert spaces.<n>We show that a novel form of representer theorem emerges: a family of transformed kernels can be defined via a system of simultaneous integral equations.<n>Remarkably, the dual coefficients are all analytically fixed to unity, obviating the need to solve a costly optimization problem to obtain the dual coefficients.
  arXiv  Detail & Related papers  (2025-10-10T02:00:56Z)
• Generalization Bound of Gradient Flow through Training Trajectory and Data-dependent Kernel [55.82768375605861]
  We establish a generalization bound for gradient flow that aligns with the classical Rademacher complexity for kernel methods.<n>Unlike static kernels such as NTK, the LPK captures the entire training trajectory, adapting to both data and optimization dynamics.
  arXiv  Detail & Related papers  (2025-06-12T23:17:09Z)
• Log-Linear Attention [81.09631871212211]
  This paper develops log-linear attention, an attention mechanism that balances linear attention's efficiency and the expressiveness of softmax attention.<n>We show that with a particular growth function, log-linear attention admits a similarly matmul-rich parallel form whose compute cost is log-linear in sequence length.<n>Log-linear attention is a general framework and can be applied on top of existing linear attention variants.
  arXiv  Detail & Related papers  (2025-06-05T08:44:51Z)
• CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interaction [77.8576094863446]
  We propose a new detextbfCoupled dutextbfAl-interactive lineatextbfR atttextbfEntion (CARE) mechanism. We first propose an asymmetrical feature decoupling strategy that asymmetrically decouples the learning process for local inductive bias and long-range dependencies. By adopting a decoupled learning way and fully exploiting complementarity across features, our method can achieve both high efficiency and accuracy.
  arXiv  Detail & Related papers  (2024-11-25T07:56:13Z)
• Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences [60.489682735061415]
  We propose CHELA, which replaces state space models with short-long convolutions and implements linear attention in a divide-and-conquer manner. Our experiments on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-06-12T12:12:38Z)
• Stable Nonconvex-Nonconcave Training via Linear Interpolation [51.668052890249726]
  This paper presents a theoretical analysis of linearahead as a principled method for stabilizing (large-scale) neural network training. We argue that instabilities in the optimization process are often caused by the nonmonotonicity of the loss landscape and show how linear can help by leveraging the theory of nonexpansive operators.
  arXiv  Detail & Related papers  (2023-10-20T12:45:12Z)
• FLatten Transformer: Vision Transformer using Focused Linear Attention [80.61335173752146]
  Linear attention offers a much more efficient alternative with its linear complexity. Current linear attention approaches either suffer from significant performance degradation or introduce additional computation overhead. We propose a novel Focused Linear Attention module to achieve both high efficiency and expressiveness.
  arXiv  Detail & Related papers  (2023-08-01T10:37:12Z)
• Linear Self-Attention Approximation via Trainable Feedforward Kernel [77.34726150561087]
  In pursuit of faster computation, Efficient Transformers demonstrate an impressive variety of approaches. We aim to expand the idea of trainable kernel methods to approximate the self-attention mechanism of the Transformer architecture.
  arXiv  Detail & Related papers  (2022-11-08T08:14:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.