P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8

• URL: http://arxiv.org/abs/2606.06521v1
• Date: Tue, 02 Jun 2026 17:29:15 GMT
• Title: P-Cast Precision in FP8 Attention: Sink-Induced Collapse and the Optimality of S=2^8
• Abstract summary: We analyze two implementation choices that affect output precision under the Attention Sink phenomenon.<n>We show that forward KV iteration causes "P-collapse" -- to leading order, a fraction Phi(Delta + delta_k - 6.93 - ln S) of non-sink P values underflow to zero.<n>We also give a constructive characterization of S = 256 = 28 as the static scale that simultaneously satisfies bit-exact IEEE 754 scaling.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: FP8 (E4M3) acceleration for attention computation offers significant throughput gains, but the 3-bit mantissa introduces precision challenges when the softmax probability matrix P is cast to FP8 before the P*V matrix multiplication. We analyze two implementation choices that affect output precision under the Attention Sink phenomenon: (1) the KV block iteration order, and (2) the static scaling factor applied to P before casting. We show that forward KV iteration causes "P-collapse" -- to leading order, a fraction Phi(Delta + delta_k - 6.93 - ln S) of non-sink P values underflow to zero, where the small shift delta_k ~ 1 (for k_sink = 4) is the expected within-sink-block score maximum -- and that reverse iteration removes it, with a zero-underflow guarantee when reverse is combined with S = 256. We further give a constructive characterization of S = 256 = 2^8 as the static scale that simultaneously satisfies (i) bit-exact IEEE 754 scaling, (ii) the lower envelope of a sawtooth function dp(S) over the E4M3 number line (dp = 2^-4, the minimum worst-case quantization step), and (iii) the maximum normal-range coverage among bit-exact (2^k) scales (a non-bit-exact scale such as 448 attains slightly higher coverage). Both optimizations are already deployed in FlashAttention-3/4 on engineering grounds; our contribution is a quantitative account of why these choices are good and a closed-form threshold Delta_c = 6.93 + ln S - delta_k for predicting kernel-level precision loss. Kernel-faithful experiments (Q, K, V in FP32 to isolate the P-cast effect) show 3-10x MSE improvement at moderate sink strengths, and paired tests confirm both fixes saturate to the same precision floor when combined.

Related papers

• Mixed-Precision SEM-Based CFD Simulations on GPUs: A Taylor-Green Vortex case [2.599882743586164]
  We study Taylor-Green vortex case using Neko, a matrix-free CFD solver based on the spectral element method (SEM)<n> Profiling shows that the fluid time step is not dominated by Krylov convergence alone.<n>We propose a three-level hierarchical mixed-precision control model.
  arXiv  Detail & Related papers  (2026-08-25T10:06:48Z)
• Circuit Depth Compression via Spectral Gap Amplification in Quantum Phase Estimation [1.2318267573115809]
  We show that quantum phase estimation can be significantly compressed in depth by preprocessing the input operator.<n>We prove that this compression is exact, bounded by log2(1/(4w Delta_lambda) + 1, and impossible for exactly spectra.<n>A net resource advantage occurs when 4w2 (2Delta_m - 1) > Delta_lambda log (1/epsilon)
  arXiv  Detail & Related papers  (2026-08-14T06:45:48Z)
• CurveFP: Co-Designing Numerical Representation and Product Arithmetic for Language Models [0.0]
  We introduce CurveFP, a block-scaled family that distributes magnitudes across interleaved logarithmic curves.<n>We instantiate the algebra as CurveFP8 E4C3/E5C2 for training and CurveFP7 E3C3 for compact inference.<n>On four 7B-9B models, CurveFP7 beats tensorwise FP8 perplexity with one fewer element bit and stays within 1.32% of native quality.
  arXiv  Detail & Related papers  (2026-08-08T01:55:46Z)
• HyperQuant: A Rate-Distortion-Optimal Quantization Pipeline for Large Language and Diffusion Models [0.0]
  HyperQuant is a unified post-training quantization pipeline for the weights and the KV cache of large language and diffusion transformers.<n>It outperforms the recent HIGGS scheme at every operating point from 3 to 5 bits per scalar (bps) on weights, and beats both TurboQuant and OCTOPUS on KV quantization down to 1.7 bps.<n>It compresses the linear weights 3.9x and the KV cache 3.79x at near-lossless quality.
  arXiv  Detail & Related papers  (2026-06-22T14:30:47Z)
• A Penalty-Free Pipeline for Direct Quantum-Annealer Portfolio Optimization [0.0]
  Direct quantum-annealer portfolio optimization is commonly formulated as a penalty-encoded QUBO submitted to D-Wave hardware.<n>We show that this standard formulation fails on current devices and identify the structural reason.<n>We propose dropping the penalty entirely: build an objective-only QUBO from the expected returns and sample it on hardware.
  arXiv  Detail & Related papers  (2026-05-17T19:50:04Z)
• Universal Transformers Need Memory: Depth-State Trade-offs in Adaptive Recursive Reasoning [0.0]
  We study learned memory tokens as computational scratchpad for a single-block Universal Transformer (UT) on SudokuExtreme.<n>We find that memory tokens are empirically necessary: across all configurations tested, no configuration without memory tokens non-trivial performance.
  arXiv  Detail & Related papers  (2026-04-23T18:30:01Z)
• When Flat Minima Fail: Characterizing INT4 Quantization Collapse After FP32 Convergence [0.0]
  Post-training quantization assumes that a well-converged model is a quantization-ready model.<n>We show this assumption fails in a structured, measurable, and previously uncharacterized way.
  arXiv  Detail & Related papers  (2026-04-16T15:46:36Z)
• Quantization Dominates Rank Reduction for KV-Cache Compression [0.0]
  quantization consistently outperforms rank reduction by 4-364 PPL depending on model and compression level.<n>We formalize this via a result showing projection damage exceeds quantization damage by 3 x 2 (2b) per direction under the softmax Fisher metric.
  arXiv  Detail & Related papers  (2026-04-13T14:06:18Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• HAS-VQ: Hessian-Adaptive Sparse Vector Quantization for High-Fidelity LLM Compression [0.0]
  We introduce HAS-VQ (Hessian-Adaptive Sparse Vec- tor Quantization), a compression framework that strictly decouples high-sensitivity outliers from the bulk weight distribution.<n>We evaluate HAS-VQ on SmolLM2-1.7B, demonstrating two distinct regimes of superiority.
  arXiv  Detail & Related papers  (2026-01-11T15:35:10Z)
• ECPv2: Fast, Efficient, and Scalable Global Optimization of Lipschitz Functions [91.31596562378076]
  ECPv2 is a scalable and theoretically grounded algorithm for global optimization of Lipschitz-continuous functions with unknown Lipschitzs.<n>We show that ECPv2 retains ECP's no-regret guarantees with optimal-time bounds and expands the acceptance region with these findings.
  arXiv  Detail & Related papers  (2025-11-20T17:30:55Z)
• Speeding Up MACE: Low-Precision Tricks for Equivarient Force Fields [51.95157731126864]
  Machine-learning force fields can deliver accurate molecular dynamics (MD) at high computational cost.<n>This thesis aims to make MACE cheaper and faster by identifying computational bottlenecks and evaluating low-precision execution policies.
  arXiv  Detail & Related papers  (2025-10-23T14:02:34Z)
• The case for 4-bit precision: k-bit Inference Scaling Laws [75.4335600212427]
  Quantization methods reduce the number of bits required to represent each parameter in a model. The final model size depends on both the number of parameters of the original model and the rate of compression. We run more than 35,000 zero-shot experiments with 16-bit inputs and k-bit parameters to examine which quantization methods improve scaling for 3 to 8-bit precision.
  arXiv  Detail & Related papers  (2022-12-19T18:48:33Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)
• HAWQV3: Dyadic Neural Network Quantization [73.11579145354801]
  Current low-precision quantization algorithms often have the hidden cost of conversion back and forth from floating point to quantized integer values. We present HAWQV3, a novel mixed-precision integer-only quantization framework.
  arXiv  Detail & Related papers  (2020-11-20T23:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.