Search Your Block Floating Point Scales!

• URL: http://arxiv.org/abs/2605.12464v1
• Date: Tue, 12 May 2026 17:50:08 GMT
• Title: Search Your Block Floating Point Scales!
• Abstract summary: Quantization is a technique for accelerating inference for generative models.<n>Recently, GPU accelerators have added first-class support for microscaling formats.<n>We propose ScaleSearch to minimize the quantization error for the given distribution.
• Score: 35.789942702223414
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantization has emerged as a standard technique for accelerating inference for generative models by enabling faster low-precision computations and reduced memory transfers. Recently, GPU accelerators have added first-class support for microscaling Block Floating Point (BFP) formats. Standard BFP algorithms use a fixed scale based on the maximum magnitude of the block. We observe that this scale choice can be suboptimal with respect to quantization errors. In this work, we propose ScaleSearch, an alternative strategy for selecting these scale factors: using a fine-grained search leveraging the mantissa bits in microscaling formats to minimize the quantization error for the given distribution. ScaleSearch can be integrated with existing quantization methods such as Post Training Quantization and low-precision attention, and is shown to improve their performance. Additionally, we introduce ScaleSearchAttention, an accelerated NVFP4-based attention algorithm, which uses ScaleSearch and adapted prior techniques to ensure near-0 performance loss for causal language modeling. Experiments show that ScaleSearch reduces quantization error by 27% for NVFP4 and improves language model PTQ by up to 15 points for MATH500 (Qwen3-8B), while ScaleSearchAttention improves Wikitext-2 PPL by upto 0.77 points for Llama 3.1 70B. The proposed methods closely match baseline performance while providing quantization accuracy improvements.

Related papers

• Optimal Post-Training Quantization Scales and Where to Find Them [1.3048920509133808]
  Post-training quantization (PTQ) compresses large language models by mapping weights to low representations.<n>We present PiSO, an algorithm that computes the optimal channel-wise weight scales exactly and efficiently under round-to-nearest quantization.
  arXiv  Detail & Related papers  (2026-06-09T14:03:04Z)
• Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models [50.16014098038291]
  Post-training quantization (PTQ) is critical for the efficient deployment of large language models (LLMs)<n>We propose SAGE-PTQ, a novel ultra-low-bit quantization framework for LLMs that minimizes hidden scaling cost.<n>On LLaMA-3B, SAGE-PTQ achieves 6.74 WikiText2 perplexity, compared to 55.8 for BiLLM, while using less than 50% of BiLLM's GPU memory.
  arXiv  Detail & Related papers  (2026-06-03T20:51:52Z)
• ScaleSweep: Accurate NVFP4 Post-Training Quantization of LLMs via Block Scale Initialization [40.425536766574844]
  We propose ScaleSweep as a simple and efficient scale optimization method for NVFP4 quantization.<n> Experiments on Llama and Qwen models demonstrate that ScaleSweep consistently improves quantization performance.<n>In particular, under aggressive end-to-end quantization of weights, activations, KV cache, and query states, ScaleSweep preserves more than 93% of the full-precision performance.
  arXiv  Detail & Related papers  (2026-05-30T08:26:16Z)
• SOAR: Scale Optimization for Accurate Reconstruction in NVFP4 Quantization [19.022444007775896]
  NVFP4 has recently emerged as an efficient 4-bit microscaling format for large language models.<n>Existing methods often yield suboptimal performance due to inflexible scale selection and the coupled treatment of quantization and dequantization scales.<n>We propose Scale Optimization for Accurate Reconstruction (SOAR), a novel post-training quantization framework that improves the accuracy of NVFP4 quantization.
  arXiv  Detail & Related papers  (2026-05-12T15:13:18Z)
• Quantization Range Estimation for Convolutional Neural Networks [12.047887178191134]
  We present a range estimation method to improve the quantization performance for post-training quantization.<n>Our experiments demonstrate that our method outperforms state-of-the-art performance generally on top-1 accuracy for image classification tasks.
  arXiv  Detail & Related papers  (2025-10-05T05:35:12Z)
• Fine-tuning Quantized Neural Networks with Zeroth-order Optimization [21.0540879091664]
  We propose Quantized Zeroth-order Optimization (QZO), a simple yet effective approach that perturbs the continuous quantization scale for gradient estimation.<n>QZO can reduce the total memory cost by more than 18$times$ for 4-bit LLMs, and enables fine-tuning Llama-2-13B within a single 24GB GPU.
  arXiv  Detail & Related papers  (2025-05-19T17:55:15Z)
• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models [64.84734437930362]
  Large Language Models (LLMs) suffer severe performance degradation when facing extremely low-bit (sub 2-bit) quantization.<n>We propose an extremely low-bit PTQ method called PTQ1.61, which enables weight quantization to 1.61-bit for the first time.<n>Experiments indicate our PTQ1.61 achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2025-02-18T08:04:58Z)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models [63.118592279833656]
  Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs)<n>We propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise.<n> Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths.
  arXiv  Detail & Related papers  (2024-05-23T16:21:48Z)
• FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search [50.07268323597872]
  We propose the first one-shot mixed-precision quantization search that eliminates the need for retraining in both integer and low-precision floating point models. With integer models, we increase the accuracy of ResNet-18 on ImageNet by 1.31% and ResNet-50 by 0.90% with equivalent model cost over previous methods. For the first time, we explore a novel mixed-precision floating-point search and improve MobileNetV2 by up to 0.98% compared to prior state-of-the-art FP8 models.
  arXiv  Detail & Related papers  (2023-08-07T04:17:19Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• Multi-objective Recurrent Neural Networks Optimization for the Edge -- a Quantization-based Approach [2.1987431057890467]
  This article introduces a Multi-Objective Hardware-Aware Quantization (MOHAQ) method, which considers both hardware efficiency and inference error as objectives for mixed-precision quantization. We propose a search technique named "beacon-based search" to retrain selected solutions only in search space and use them as beacons to know effect of retraining on other solutions.
  arXiv  Detail & Related papers  (2021-08-02T22:09:12Z)
• Balancing Rates and Variance via Adaptive Batch-Size for Stochastic Optimization Problems [120.21685755278509]
  In this work, we seek to balance the fact that attenuating step-size is required for exact convergence with the fact that constant step-size learns faster in time up to an error. Rather than fixing the minibatch the step-size at the outset, we propose to allow parameters to evolve adaptively.
  arXiv  Detail & Related papers  (2020-07-02T16:02:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.