Finer is Better (with the Right Scaling)

• URL: http://arxiv.org/abs/2605.08565v1
• Date: Fri, 08 May 2026 23:57:11 GMT
• Title: Finer is Better (with the Right Scaling)
• Abstract summary: A paradox recently identified in the literature demonstrates that standard abs-max scaling can actually degrade model quality as block sizes shrink.<n>We show that this degradation is driven by heavy-tailed tensor distributions interacting poorly with the coarse upper quantization bins of the FP4 element format.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Microscaling is a critical technique for preserving the quality of Large Language Models (LLMs) quantized to ultra-low precision formats. Intuitively, finer block sizes should yield lower quantization error; however, a paradox recently identified in the literature demonstrates that standard abs-max scaling can actually degrade model quality as block sizes shrink. In this work, we investigate the underlying mechanics of this phenomenon. We demonstrate that this degradation is not an inherent limitation of finer granularity, but is primarily driven by heavy-tailed tensor distributions interacting poorly with the coarse upper quantization bins of the FP4 element format. Specifically, we show that i) preventing the scaling factor from underflowing to zero mitigates localized errors, ii) targeted algorithmic interventions like the 4-over-6 methodology effectively correct the quantization geometry for large elements, and iii) a brute-force search establishes an optimal baseline, confirming that the theoretical Mean Squared Error (MSE) strictly improves with finer block sizes. Ultimately, our findings reveal a valuable interchangeability: applying the correct algorithmic recipe allows standard, hardware-compliant formats (like OCP E4M3) to match the performance of custom, wider-exponent formats (like UE5M3). We validate these results across several large language models, fully resolving the block size paradox and achieving robust downstream perplexity improvements.

Related papers

• MorphoQuant: Modality-Aware Quantization for Omni-modal Large Language Models [57.87791117741788]
  We propose a modality-aware PTQ framework to preserve cross-modal morphology and mitigate outlier loss.<n>Distribution-Aware Bias Compensation (DABC) selectively absorbs long-tailed outliers into channel-wise biases.<n>We also propose Morphology-Directed Quantization Function Optimization (MDQFO) to co-optimize the quantization grid with the bias mask.
  arXiv  Detail & Related papers  (2026-06-03T02:05:10Z)
• Efficient Long-Context Modeling in Diffusion Language Models via Block Approximate Sparse Attention [58.484607892210015]
  Diffusion Language Models (DLMs) enable globally coherent, bidirectional, and controllable text generation.<n>Many existing block-sparse attention methods select blocks by fixed sampling patterns over the high-resolution attention space.<n>We propose the Block Approximate Sparse Attention framework (BA-Att) with block-wise pre-downsampled operation.
  arXiv  Detail & Related papers  (2026-05-19T12:01:13Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• Is Finer Better? The Limits of Microscaling Formats in Large Language Models [6.978989632396424]
  We report the emergence of a surprising behavior associated with microscaling quantization.<n>The output of a quantized model degrades as block size is decreased below a given threshold.<n>We propose the use of FP8 unsigned E5M3 as a novel hardware-friendly format for the scales in FP4 microscaling data types.
  arXiv  Detail & Related papers  (2026-01-26T23:21:24Z)
• Improving Block-Wise LLM Quantization by 4-bit Block-Wise Optimal Float (BOF4): Analysis and Variations [22.127873567034825]
  Large language models (LLMs) demand extensive memory capacity during both fine-tuning and inference.<n>Existing methods apply block-wise quantization techniques, such as NF4 and AF4, to the network weights.<n>We show that these quantization techniques incur suboptimal quantization errors.
  arXiv  Detail & Related papers  (2025-05-10T14:00:15Z)
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem [71.3332971315821]
  We present a "line theoremarity" establishing a direct relationship between the layer-wise $ell$ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels.
  arXiv  Detail & Related papers  (2024-11-26T15:35:44Z)
• Mitigating the Impact of Outlier Channels for Language Model Quantization with Activation Regularization [62.15918574997175]
  It is known that language models contain outlier channels whose values on average are orders of magnitude higher than other channels. We propose a strategy which regularizes a layer's inputs via quantization-aware training (QAT) and its outputs via activation kurtosis regularization. We show that regularizing both the inputs and outputs is crucial for preventing a model's "migrating" the difficulty in input quantization to the weights.
  arXiv  Detail & Related papers  (2024-04-04T17:25:30Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• NF4 Isn't Information Theoretically Optimal (and that's Good) [0.38073142980733]
  I show that this can't quite be the case, as the distribution of the values to be quantized depends on the block-size. I attempt to apply these insights to derive an improved code based on minimizing the expected L1 reconstruction error, rather than the quantile based method.
  arXiv  Detail & Related papers  (2023-06-12T08:52:14Z)
• Block Format Error Bounds and Optimal Block Size Selection [7.056118133284956]
  One of the most promising and rapidly advancing frontiers here is the creation of new data formats. We focus on the family of block floating point numerical formats due to their combination of wide dynamic range, numerical accuracy, and efficient hardware implementation of inner products using simple integer arithmetic.
  arXiv  Detail & Related papers  (2022-10-11T14:15:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.