Hardware-Aware FP4 FlashAttention-4

• URL: http://arxiv.org/abs/2609.04105v1
• Date: Thu, 03 Sep 2026 17:12:35 GMT
• Title: Hardware-Aware FP4 FlashAttention-4
• Abstract summary: Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200.<n>Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.
• Score: 4.560316459247797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Blackwell's 4-bit floating-point (FP4) tensor cores do not automatically make attention faster because softmax conversion and on-chip dependencies dominate once its matrix products shrink. We address this with \emph{Direct-P} for noncausal inference and a causal path that passes the forward quantization directly into backward. Direct-P maps scores directly to FP4 probabilities and reaches up to 2.13$\times$ the bfloat16 (BF16) forward throughput on an NVIDIA GB200. The causal path reconstructs probabilities from saved quantized queries and keys and uses 8-bit floating-point (FP8) gradient operands, accelerating a complete single-GPU 8-billion-parameter update by up to 1.14$\times$. Matched distributed training retains FP8 probabilities and values; every tested MXFP4 probability/value training trajectory diverges.

Related papers

• VC-Attention: Value Smoothing and Softmax Casting for Low-bit Attention [47.30761996669105]
  We propose VC-Attention, a training-free low-bit attention framework.<n>It addresses both by smoothing value tokens by a probability Cast. V-Smooth reorders value tokens by lightweight clustering online, so the tokens in a fused hardware block quantize well together.<n>It maps log-domain scores directly to E4M3 probability codes with one fused multiply-add, eliminating the FP32 exponential and the format conversion.
  arXiv  Detail & Related papers  (2026-09-14T16:17:35Z)
• EFQ-Softmax: Exp-Free Quantization for Softmax [3.5456060266135734]
  Low-bit attention accelerates Transformer inference by moving $QKtop$ and $PV$ matrix multiplications to FP8 or FP4 matrix engines.<n>The softmax path often evaluates shifted-score exponentials in higher precision, forms a temporary probability block, and quantizes it before low-bit $PV$ multiplication.<n>We propose EFQ-Softmax, a low-bit probability-generation method that directly maps shifted attention scores to block-scaled E2M1 operands.
  arXiv  Detail & Related papers  (2026-09-09T05:04:33Z)
• LongLive-2.0: An NVFP4 Parallel Infrastructure for Long Video Generation [84.86814271505109]
  LongLive-2.0 is an NVFP4-based parallel infrastructure throughout the full training and inference workflow of long video generation.<n>For training, we introduce sequence-parallel autoregressive (AR) training, instantiated as Balanced SP.<n>Experiments show up to 2.15x speedup in training, and 1.84x in inference.
  arXiv  Detail & Related papers  (2026-05-18T17:57:03Z)
• Pretraining large language models with MXFP4 on Native FP4 Hardware [6.139566055770847]
  We study why full-pipeline FP4 training of large language models often diverges, even when forward activations and activation gradients remain stable.<n>We find that FP4 training instability is driven by structured micro-scaling errors along sensitive gradient paths, rather than by insufficient Instinctity.
  arXiv  Detail & Related papers  (2026-05-11T00:04:07Z)
• SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity [86.71343842875878]
  NVIDIA's 2:4 Sparse Cores deliver 2x throughput but demand strict 50% pruning.<n>Milder $(2N-2):2N$ patterns preserve accuracy yet receive no hardware support.<n>We present SlideSparse, the first system to unlock Sparse Core acceleration.
  arXiv  Detail & Related papers  (2026-03-05T14:49:16Z)
• Practical FP4 Training for Large-Scale MoE Models on Hopper GPUs [2.576131688630686]
  Training large-scale Mixture-of-Experts (MoE) models is bottlenecked by activation memory and expert-parallel communication.<n>We present a training recipe that enables MXFP4 efficiency for MoE models on Hopper without native 4-bit support.
  arXiv  Detail & Related papers  (2026-03-03T08:29:19Z)
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling [13.357423392911036]
  We introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values.<n>We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform.<n>We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy.
  arXiv  Detail & Related papers  (2025-12-01T18:59:45Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields [51.95157731126864]
  Machine-learning force fields can deliver accurate molecular dynamics (MD) at high computational cost.<n>This thesis aims to make MACE cheaper and faster by identifying computational bottlenecks and evaluating low-precision execution policies.
  arXiv  Detail & Related papers  (2025-10-23T14:02:34Z)
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization [77.67818998672516]
  We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization.<n>We introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm.<n>We show that MR-GPTQ matches or outperforms state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-09-27T09:22:21Z)
• Training LLMs with MXFP4 [16.524414449291488]
  We present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2times$ faster than FP8 on supported hardware.<n>Our recipe computes $>1/2$ the training FLOPs in MXFP4, enabling an estimated speedup of $>1.3times$ over FP8 and $>1.7times$ over BF16 during backpropagation.
  arXiv  Detail & Related papers  (2025-02-27T23:01:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.