Replacing Softmax Similarity with a Sharpened Angular Similarity: Theory and Practice of Scaling To Billion-Context Attention

• URL: http://arxiv.org/abs/2510.04008v2
• Date: Thu, 23 Oct 2025 01:09:14 GMT
• Title: Replacing Softmax Similarity with a Sharpened Angular Similarity: Theory and Practice of Scaling To Billion-Context Attention
• Abstract summary: We introduce RACE Attention, a kernel-inspired alternative to Softmax Attention.<n> RACE Attention replaces the exponential kernel with a sharpened angular (cosine) similarity.<n>In a controlled scale test, it processes up to 12 million tokens during a single forward-backward pass on an NVIDIA GH200 GPU.
• Score: 21.00061869921723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Softmax Attention has a quadratic time complexity, which becomes prohibitive to run at long contexts, even with highly optimized GPU kernels. For example, FlashAttention (an exact, GPU-optimized implementation of Softmax Attention) cannot complete a single forward-backward pass of a multi-head attention layer once the context exceeds ~4 million tokens on an NVIDIA GH200 (96 GB). We introduce RACE Attention, a kernel-inspired alternative to Softmax Attention that is linear in sequence length and embedding dimension. RACE Attention replaces the exponential kernel with a sharpened angular (cosine) similarity, and approximates attention outputs via randomized projections and soft Locality-Sensitive Hashing (LSH). Across language modeling, masked language modeling, and text classification, RACE Attention matches the accuracy of strong baselines while reducing runtime and memory. In a controlled scale test, it processes up to 12 million tokens during a single forward-backward pass on an NVIDIA GH200 GPU and 75 million tokens on an Intel Xeon Gold 5220R CPU, well beyond the practical limits of the current state-of-the-art attention implementations. RACE Attention thus offers a practical, theoretically grounded mechanism for outrageously long context windows on today's hardware. We hope that it gets adopted in practice.

Related papers

• VLZip: Unified Visual and Textual Compression for Interleaved Long-Context Modeling [81.92510561363197]
  We introduce VLZip, a framework that unifies visual and textual compression for high-fidelity reasoning within a pure Transformer.<n>We also introduce LongVLBench, a new benchmark derived from video narratives that demands holistic, narrative-level reasoning.
  arXiv  Detail & Related papers  (2026-08-09T10:31:03Z)
• MiniMax Sparse Attention [16.750980846187343]
  MiniMax Sparse Attention (MSA) is a blockwise sparse attention built upon Grouped Query Attention (GQA)<n>A lightweight Index Branch scores key-value blocks and independently selects a Top-k subset for each GQA group.<n>The Main Branch then performs exact block-sparse attention over only the selected blocks.
  arXiv  Detail & Related papers  (2026-06-11T14:23:41Z)
• LoSA: Locality Aware Sparse Attention for Block-Wise Diffusion Language Models [49.93891888238178]
  Block-wise diffusion language models (DLMs) generate multiple tokens in any order, offering a promising alternative to the autoregressive decoding pipeline.<n>Naive sparse attention fails on DLMs due to a KV Inflation problem, where different queries select different prefix positions.<n>We propose LOSA (Locality-aware Sparse Attention), which reuses cached prefix-attention results for stable tokens and applies sparse attention only to active tokens.
  arXiv  Detail & Related papers  (2026-04-13T20:53:51Z)
• Diagonal-Tiled Mixed-Precision Attention for Efficient Low-Bit MXFP Inference [29.72164316945472]
  Transformer-based large language models (LLMs) have demonstrated remarkable performance across a wide range of real-world tasks.<n>But their inference cost remains prohibitively high due to the quadratic complexity of attention and the memory bandwidth limitations of high-precision operations.<n>We present a low-bit mixed-precision attention kernel using the microscaling floating-point (MXFP) data format.
  arXiv  Detail & Related papers  (2026-04-05T03:56:21Z)
• Neural Attention Search Linear: Towards Adaptive Token-Level Hybrid Attention Models [7.961563754693873]
  We propose a framework that applies both linear attention and softmax attention operations within the same layer on different tokens.<n>NAtS-L automatically determines whether a token can be handled by a linear attention model, i.e., tokens that have only short-term impact.<n>By searching for optimal Gated DeltaNet and softmax attention combinations across tokens, we show that NAtS-L provides a strong yet efficient token-level hybrid architecture.
  arXiv  Detail & Related papers  (2026-02-03T16:02:50Z)
• Spava: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention [63.69228529380251]
  Spava is a sequence-parallel framework with optimized attention for long-video inference.<n>Spava delivers speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss.
  arXiv  Detail & Related papers  (2026-01-29T09:23:13Z)
• NGPU-LM: GPU-Accelerated N-Gram Language Model for Context-Biasing in Greedy ASR Decoding [54.88765757043535]
  This work rethinks data structures for statistical n-gram language models to enable fast and parallel operations for GPU-optimized inference.<n>Our approach, named NGPU-LM, introduces customizable greedy decoding for all major ASR model types with less than 7% computational overhead.<n>The proposed approach can eliminate more than 50% of the accuracy gap between greedy and beam search for out-of-domain scenarios while avoiding significant slowdown caused by beam search.
  arXiv  Detail & Related papers  (2025-05-28T20:43:10Z)
• MonarchAttention: Zero-Shot Conversion to Fast, Hardware-Aware Structured Attention [10.244490009712466]
  We propose a novel approach to sub-quadratic attention approximation via Monarch matrices.<n>MonarchAttention is both transferable, yielding minimal performance loss with no additional training, and hardware-efficient.<n>We demonstrate the quality of MonarchAttention on diverse tasks and architectures in vision and language problems.
  arXiv  Detail & Related papers  (2025-05-24T13:44:44Z)
• MagicPIG: LSH Sampling for Efficient LLM Generation [41.75038064509643]
  We show that TopK attention itself suffers from quality degradation in certain downstream tasks because attention is not always as sparse as expected.<n>We propose MagicPIG, a heterogeneous system based on Locality Sensitive Hashing (LSH)<n>MagicPIG significantly reduces the workload of attention while preserving high accuracy for diverse tasks.
  arXiv  Detail & Related papers  (2024-10-21T16:44:51Z)
• S2-Attention: Hardware-Aware Context Sharding Among Attention Heads [49.1454481007861]
  Sparse attention selectively attends to a subset of tokens in the context.<n>It remains unclear whether sparse attention can maintain the model's quality at a scale of today's large language models.<n>This paper presents Sparsely-Sharded(S2) Attention, a Triton library that provides kernel optimization for sparse attention customizable at both per-head and per-context-range levels.
  arXiv  Detail & Related papers  (2024-07-25T00:27:07Z)
• A Training-free Sub-quadratic Cost Transformer Model Serving Framework With Hierarchically Pruned Attention [43.211427581302715]
  We propose Hierarchically Pruned Attention (HiP) to increase context length in large language models.<n>HiP reduces the time complexity of the attention mechanism to $O(T log T)$ and the space complexity to $O(T)$, where $T$ is the sequence length.<n>We show that HiP significantly reduces both prefill and decoding latencies, as well as memory usage, while maintaining high-quality generation with minimal degradation.
  arXiv  Detail & Related papers  (2024-06-14T08:32:45Z)
• Hardware-Aware Parallel Prompt Decoding for Memory-Efficient Acceleration of LLM Inference [23.633481089469836]
  Auto-regressive decoding of Large Language Models (LLMs) results in significant overheads in their hardware performance.<n>We propose a novel parallel prompt decoding that requires only $0.0002$% trainable parameters, enabling efficient training on a single A100-40GB GPU in just 16 hours.<n>Our approach demonstrates up to 2.49$times$ speedup and maintains a minimal memory overhead of just $0.0004$%.
  arXiv  Detail & Related papers  (2024-05-28T22:19:30Z)
• EVEREST: Efficient Masked Video Autoencoder by Removing Redundant Spatiotemporal Tokens [57.354304637367555]
  We present EVEREST, a surprisingly efficient MVA approach for video representation learning. It finds tokens containing rich motion features and discards uninformative ones during both pre-training and fine-tuning. Our method significantly reduces the computation and memory requirements of MVA.
  arXiv  Detail & Related papers  (2022-11-19T09:57:01Z)
• cosFormer: Rethinking Softmax in Attention [60.557869510885205]
  kernel methods are often adopted to reduce the complexity by approximating the softmax operator. Due to the approximation errors, their performances vary in different tasks/corpus and suffer crucial performance drops. We propose a linear transformer called cosFormer that can achieve comparable or better accuracy to the vanilla transformer.
  arXiv  Detail & Related papers  (2022-02-17T17:53:48Z)
• SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning [10.981433334942476]
  We present SpAtten, an efficient algorithm-architecture co-design that leverages token sparsity, head sparsity, and quantization opportunities to reduce the attention computation and memory access. Experiments on 30 benchmarks show that, on average, SpAtten reduces DRAM access by 10.0x with no accuracy loss, and achieves 1.6x, 3.0x, 162x, 347x speedup, and 1,4x, 3.2x, 1193x, 4059x energy savings.
  arXiv  Detail & Related papers  (2020-12-17T18:59:07Z)
• SMYRF: Efficient Attention using Asymmetric Clustering [103.47647577048782]
  We propose a novel type of balanced clustering algorithm to approximate attention. SMYRF can be used as a drop-in replacement for dense attention layers without any retraining. We show that SMYRF can be used interchangeably with dense attention before and after training.
  arXiv  Detail & Related papers  (2020-10-11T18:49:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.