ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling

• URL: http://arxiv.org/abs/2606.13233v1
• Date: Thu, 11 Jun 2026 11:47:40 GMT
• Title: ReSET: Accurate Latency-Critical NVFP4 Reasoning via Step-Aware Temperature Scaling
• Abstract summary: We analyze the effect of NVFP4 quantization on token-level uncertainty during reasoning.<n>We show that quantization increases incorrect sampling at low-entropy symbolic tokens.<n>We further design a core-$M$FP4 kernel for latency-critical decoding using both token-level and step-level entropy signals.
• Score: 9.642832832617144
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large reasoning models (LRMs) improve complex problem-solving by generating long intermediate reasoning traces, but this substantially increases inference costs. NVFP4 inference offers a promising approach to reduce both computational and memory costs through hardware-supported low-precision execution. However, directly applying NVFP4 to LRMs introduces two practical limitations: reasoning accuracy degrades under quantization, and existing NVFP4 kernels do not fully realize latency benefits in small-batch autoregressive decoding. In this work, we analyze the effect of NVFP4 quantization on token-level uncertainty during reasoning. We show that quantization increases incorrect sampling at low-entropy symbolic tokens, while causing over-concentration on a small set of tokens in high-uncertainty reasoning steps. Based on this observation, we propose \textbf{ReSET}, a reasoning-step entropy-based temperature-scaling method that estimates step-level uncertainty online and adapts the decoding temperature using both token-level and step-level entropy signals. To address the latency gap, we further design a CUDA-core small-$M$ NVFP4 kernel for latency-critical autoregressive decoding. Across reasoning benchmarks and model scales, ReSET improves NVFP4 reasoning accuracy by up to $\sim\!$2 points over the NVFP4 baseline. Our CUDA-core small-$M$ kernel further improves latency-critical decoding, delivering up to $2.5\!\times$ kernel-level speedup over NVFP4 vLLM and approximately $2\!\times$ end-to-end decoding speedup over BF16. Code is available at https://github.com/aiha-lab/ReSET.

Related papers

• Full-Stack FP4: Stable LLM Pretraining with Quantized Projections, Optimizers, and Attention [34.07319488267708]
  Recent NVFP4 pretraining methods mainly target transformer linear layers, leaving computation states, arithmetic and attention underexplored in 4-bit pipelines.<n>This critical gap blocks stable full-stack 4-bit pretraining, as the three core modules exhibit unique numerical failure patterns.<n>We propose Full-Stack FP4, the first complete NVFP4 pretraining framework resolving all three stability bottlenecks.
  arXiv  Detail & Related papers  (2026-07-05T17:31:36Z)
• SPEAR: A System for Post-Quantization Error-Adaptive Recovery Enabling Efficient Low-Bit LLM Serving [26.96887030437247]
  We present SPEAR, a system for post-quantization error-adaptive recovery that improves low-bit LLM serving.<n> SPEAR introduces lightweight Error Compensators (ECs) by per-token gates and places them only at the most error-sensitive layers identified through a CKA-guided entropy-aware diagnostic.<n>We show that SPEAR recovers 56-75% of the perplexity gap between W4 and FP16 while adding less than 1% model memory overhead and maintaining latency comparable to a widely used 4-bit serving deployment.
  arXiv  Detail & Related papers  (2026-06-04T22:38:53Z)
• Mix-Quant: Quantized Prefilling, Precise Decoding for Agentic LLMs [78.01752802974855]
  We propose Mix-Quant, a simple and effective phase-aware quantization framework for fast agentic inference.<n>We show that Mix-Quant preserves task performance while delivering significant efficiency improvements, achieving up to a 3x speedup during prefilling.
  arXiv  Detail & Related papers  (2026-05-19T17:50:17Z)
• OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond [50.440302567029654]
  Multi-modal intelligence has made the Key-Value cache a dominant memory bottleneck for efficient deployment.<n>In this work, we revisit the inherent limitations of the per-channel quantization paradigm.<n>We propose OScaR, an accurate and lightweight KV cache compression framework for X-LLMs.
  arXiv  Detail & Related papers  (2026-05-19T10:53:03Z)
• BWTA: Accurate and Efficient Binarized Transformer by Algorithm-Hardware Co-design [71.97035034203275]
  We analyze zero-point distortion in binarization and propose a Binary Weights & Ternary Activations (BWTA) quantization scheme.<n>For training, we propose Smooth Multi-Stage Quantization, combining a Levelwise Degradation Strategy and a MagnitudeAlignment Projection Factor.<n>Experiments show that BWTA approaches full-precision performance for Transformer-based models, with an average 3.5% drop on GLUE and less than 2% drop on five tasks.
  arXiv  Detail & Related papers  (2026-04-05T04:25:07Z)
• Dissecting Outlier Dynamics in LLM NVFP4 Pretraining [46.10969678564592]
  This study conducts a longitudinal analysis of outlier dynamics across architecture during NVFP4 pretraining.<n>We find that, compared with Softmax Attention (SA), Linear Attention (LA) reduces per-tensor heavy tails but still exhibits persistent block-level spikes under block quantization.<n>We then develop CHON, an NVFP4 training recipe integrating with post-QK operation protection.
  arXiv  Detail & Related papers  (2026-02-02T12:50:27Z)
• Quartet II: Accurate LLM Pre-Training in NVFP4 by Improved Unbiased Gradient Estimation [40.140261007984215]
  We improve the state of the art for quantized training in NVFP4 via a novel unbiased quantization routine for micro-scaled formats.<n>We show analytically that Quartet II achieves consistently better gradient estimation across all major matrix multiplications.
  arXiv  Detail & Related papers  (2026-01-30T10:39:11Z)
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization [77.67818998672516]
  We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization.<n>We introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm.<n>We show that MR-GPTQ matches or outperforms state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-09-27T09:22:21Z)
• R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoning [80.104336426172]
  Chain-of-thought (CoT) enhances problem-solving ability of large language models.<n>CoT incurs substantial inference cost due to long autoregressive trajectories.<n>We introduce R-Stitch, a training-free hybrid decoding framework.
  arXiv  Detail & Related papers  (2025-07-23T08:14:36Z)
• Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics [64.62231094774211]
  Statefuls (e.g., Adam) maintain auxiliary information even 2x the model size in order to achieve optimal convergence.<n>SOLO enables Adam-styles to maintain quantized states with precision as low as 3 bits, or even 2 bits.<n>SOLO can thus be seamlessly applied to Adam-styles, leading to substantial memory savings with minimal accuracy loss.
  arXiv  Detail & Related papers  (2025-05-01T06:47:45Z)
• Understanding INT4 Quantization for Transformer Models: Latency Speedup, Composability, and Failure Cases [24.34969722921442]
  We show that W4A4 quantization introduces no to negligible accuracy degradation for encoder-only and encoder-decoder models, but causes a significant accuracy drop for decoder-only models. We develop a highly optimized end-to-end W4A4 encoder inference pipeline supporting different quantization strategies.
  arXiv  Detail & Related papers  (2023-01-27T22:44:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.