VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation

• URL: http://arxiv.org/abs/2604.12798v1
• Date: Tue, 14 Apr 2026 14:28:50 GMT
• Title: VFA: Relieving Vector Operations in Flash Attention with Global Maximum Pre-computation
• Abstract summary: FlashAttention-style online softmax enables exact attention computation with linear memory.<n>Non-matmul components of online softmax can become vector or SIMD limited and dominate latency.<n>This paper proposes Vector Relieved Flash Attention (VFA), a hardware-friendly method that reduces rowmax-driven updates of the running maximum.
• Score: 5.279829639786756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: FlashAttention-style online softmax enables exact attention computation with linear memory by streaming score tiles through on-chip memory and maintaining a running maximum and normalizer. However, as attention kernels approach peak tensor-core/cube-core throughput on modern accelerators, non-matmul components of online softmax -- especially per-tile rowmax and rowsum reductions and rescale chains -- can become vector or SIMD limited and dominate latency. This paper revisits FlashAttention and proposes Vector Relieved Flash Attention (VFA), a hardware-friendly method that reduces rowmax-driven updates of the running maximum while retaining the online-softmax structure. VFA initializes the running maximum via a cheap approximation from key-block representations, reorders key-block traversal to prioritize high-impact sink and local blocks, and freezes the maximum for remaining blocks to avoid repeated reductions and rescaling. We further integrate VFA with block-sparse skipping methods such as BLASST to form Vector Relieved Sparse Attention (VSA), which reduces both block count and per-block overhead. Notably, VFA and VSA completely avoid the conditional rescale operation in the update stage used in FA4.0. Extensive evaluations on benchmarks including MMLU and MATH500, together with attention statistics, verify our design: (i) sink and local reordering stabilizes the running maximum early; (ii) simple Q and K block summaries fail due to intra-block heterogeneity; (iii) m-initialization is required when maxima appear in middle blocks. Overall, VFA and VSA efficiently alleviate online-softmax reduction bottlenecks without performance loss. Compared to the C16V32 baseline, C8V32, C4V32 and C4V16 achieve nearly two times speedup on modern hardware while hitting the vector bottleneck. With upcoming architecture improvements, C4V16 will deliver six times speedup by enhancing exponent capacity.

Related papers

• VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention [47.30761996669105]
  We propose VC-Attention, a training-free low-bit attention framework.<n>It addresses both by smoothing value tokens by a probability Cast. V-Smooth reorders value tokens by lightweight clustering online, so the tokens in a fused hardware block quantize well together.<n>It maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion.
  arXiv  Detail & Related papers  (2026-09-14T16:17:35Z)
• FlashPrefill V2: Block-Sparse Prefill Attention for Long-Context LLM Serving [45.624568309628756]
  Long-context modeling is a pivotal capability for Large Language Models.<n>Our previous work, FlashPrefill, mitigates this cost through instantaneous pattern discovery and max-based dynamic thresholding.<n>FlashPrefill V2 evolves FlashPrefill from a prototype toward practical long-context serving along three dimensions.
  arXiv  Detail & Related papers  (2026-08-20T08:02:55Z)
• FlashAttention for Scalable Vector Architectures [0.05735035463793007]
  We present FlashAttention-V, a blocked FlashAttention for scalable vector architectures that adapts efficiently from short to very long vectors.<n>We show that FlashAttention-V achieves 22x-42x speedup over scalar FlashAttention at 512-bit VL in prefill, with an additional 2x-2.5x gain scaling to 64 lanes and 4096bit VL.<n>During decode, FlashAttention-V achieves 8x-11x speedup using 512-bit vector lengths over scalar FlashAttention, with performance showing diminishing sensitivity to vector width and lane count.
  arXiv  Detail & Related papers  (2026-08-19T08:02:17Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling [20.849774181776414]
  FlashAttention-4 achieves up to 1.3$times$ speedup over cuDNN 9.13 and 2.7$times$ over Triton on B200 GPUs with BF16.<n>We implement FlashAttention-4 entirely in CuTe- embedded in Python, achieving 20-30$times$ faster compile times compared to traditional C++ template-based approaches.
  arXiv  Detail & Related papers  (2026-03-05T18:24:49Z)
• Spava: Accelerating Long-Video Understanding via Sequence-Parallelism-aware Approximate Attention [63.69228529380251]
  Spava is a sequence-parallel framework with optimized attention for long-video inference.<n>Spava delivers speedups of 12.72x, 1.70x, and 1.18x over FlashAttn, ZigZagRing, and APB, without notable performance loss.
  arXiv  Detail & Related papers  (2026-01-29T09:23:13Z)
• BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding [28.11907989174509]
  We introduce BLASST, a drop-in sparse attention method that dynamically prunes the attention matrix without any pre-computation or proxy scores.<n>Our method uses a fixed threshold and existing information from online softmax to identify negligible attention scores.<n>We develop an automated calibration procedure that reveals a simple inverse relationship between optimal threshold and context length.
  arXiv  Detail & Related papers  (2025-12-12T23:30:43Z)
• InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Models [49.08289742711585]
  We propose InfiniteVL, a linear-complexity VLM architecture that synergizes sliding window attention (SWA) with Gated DeltaNet.<n>We show that InfiniteVL achieves over 3.6times inference speedup while maintaining constant latency and memory footprint.<n>In streaming video understanding scenarios, it sustains a stable 24 FPS real-time prefill speed while preserving long-term memory cache.
  arXiv  Detail & Related papers  (2025-12-09T17:18:32Z)
• InfLLM-V2: Dense-Sparse Switchable Attention for Seamless Short-to-Long Adaptation [56.694702609077495]
  Long-sequence processing is a critical capability for modern large language models.<n>InfLLM-V2 is a trainable sparse attention framework that seamlessly adapts models from short to long sequences.<n>In experiments, InfLLM-V2 is 4$times$ faster than dense attention while retaining 98.1% and 99.7% of the performance.
  arXiv  Detail & Related papers  (2025-09-29T12:08:33Z)
• SystolicAttention: Fusing FlashAttention within a Single Systolic Array [2.8650887057567864]
  Transformer models rely heavily on scaled dot-product attention (SDPA)<n>Current systolic-array-based accelerators face significant challenges when executing FlashAttention.<n>We propose FSA, an enhanced systolic array architecture that enables the FlashAttention algorithm to run entirely within a single systolic array.
  arXiv  Detail & Related papers  (2025-07-15T14:04:17Z)
• LiVOS: Light Video Object Segmentation with Gated Linear Matching [116.58237547253935]
  LiVOS is a lightweight memory network that employs linear matching via linear attention. For longer and higher-resolution videos, it matched STM-based methods with 53% less GPU memory and supports 4096p inference on a 32G consumer-grade GPU.
  arXiv  Detail & Related papers  (2024-11-05T05:36:17Z)
• vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving [53.972175896814505]
  Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests. Large Language Models (LLMs) are widely used across various domains, processing millions of daily requests.
  arXiv  Detail & Related papers  (2024-07-22T14:37:58Z)
• EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction [67.11722682878722]
  This work presents EfficientViT, a new family of high-resolution vision models with novel multi-scale linear attention. Our multi-scale linear attention achieves the global receptive field and multi-scale learning. EfficientViT delivers remarkable performance gains over previous state-of-the-art models.
  arXiv  Detail & Related papers  (2022-05-29T20:07:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.