Is Finer Better? The Limits of Microscaling Formats in Large Language Models

• URL: http://arxiv.org/abs/2601.19026v1
• Date: Mon, 26 Jan 2026 23:21:24 GMT
• Title: Is Finer Better? The Limits of Microscaling Formats in Large Language Models
• Abstract summary: We report the emergence of a surprising behavior associated with microscaling quantization.<n>The output of a quantized model degrades as block size is decreased below a given threshold.<n>We propose the use of FP8 unsigned E5M3 as a novel hardware-friendly format for the scales in FP4 microscaling data types.
• Score: 6.978989632396424
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Microscaling data formats leverage per-block tensor quantization to enable aggressive model compression with limited loss in accuracy. Unlocking their potential for efficient training and inference necessitates hardware-friendly implementations that handle matrix multiplications in a native format and adopt efficient error-mitigation strategies. Herein, we report the emergence of a surprising behavior associated with microscaling quantization, whereas the output of a quantized model degrades as block size is decreased below a given threshold. This behavior clashes with the expectation that a smaller block size should allow for a better representation of the tensor elements. We investigate this phenomenon both experimentally and theoretically, decoupling the sources of quantization error behind it. Experimentally, we analyze the distributions of several Large Language Models and identify the conditions driving the anomalous behavior. Theoretically, we lay down a framework showing remarkable agreement with experimental data from pretrained model distributions and ideal ones. Overall, we show that the anomaly is driven by the interplay between narrow tensor distributions and the limited dynamic range of the quantized scales. Based on these insights, we propose the use of FP8 unsigned E5M3 (UE5M3) as a novel hardware-friendly format for the scales in FP4 microscaling data types. We demonstrate that UE5M3 achieves comparable performance to the conventional FP8 unsigned E4M3 scales while obviating the need of global scaling operations on weights and activations.

Related papers

• Finer is Better (with the Right Scaling) [0.0]
  A paradox recently identified in the literature demonstrates that standard abs-max scaling can actually degrade model quality as block sizes shrink.<n>We show that this degradation is driven by heavy-tailed tensor distributions interacting poorly with the coarse upper quantization bins of the FP4 element format.
  arXiv  Detail & Related papers  (2026-05-08T23:57:11Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• Approaching Maximal Information Extraction in Low-Signal Regimes via Multiple Instance Learning [0.0]
  We propose a new machine learning (ML) methodology to obtain more precise predictions.<n>We show that it might be possible to extract the theoretical maximum Fisher Information latent in a dataset.
  arXiv  Detail & Related papers  (2025-08-09T22:46:55Z)
• Characterization and Mitigation of Training Instabilities in Microscaling Formats [6.025438902954768]
  Training large language models is an expensive, compute-bound process.<n>Next-generation hardware accelerators increasingly support lower-precision arithmetic formats.<n>We investigate the challenges and viability of block-scaled precision formats during model training.
  arXiv  Detail & Related papers  (2025-06-25T18:25:08Z)
• AQUATIC-Diff: Additive Quantization for Truly Tiny Compressed Diffusion Models [4.373803477995854]
  This work applies codebook-based additive vector quantization to the problem of diffusion model compression.<n>We report sFID 1.92 points lower than the full-precision model at W4A8 and the best-reported results for FID, sFID and ISC at W2A8.<n>We are also able to demonstrate FLOPs savings on arbitrary hardware via an efficient inference kernel.
  arXiv  Detail & Related papers  (2025-06-06T10:37:09Z)
• Unified Scaling Laws for Compressed Representations [69.72517034565467]
  We investigate whether a unified scaling framework can accurately predict model performance when training occurs over various compressed representations.<n>Our main finding is demonstrating both theoretically and empirically that there exists a simple "capacity" metric.<n>We extend our formulation to directly compare the accuracy potential of different compressed formats, and to derive better algorithms for training over sparse-quantized formats.
  arXiv  Detail & Related papers  (2025-06-02T16:52:51Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem [71.3332971315821]
  We present a "line theoremarity" establishing a direct relationship between the layer-wise $ell$ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels.
  arXiv  Detail & Related papers  (2024-11-26T15:35:44Z)
• Error Diffusion: Post Training Quantization with Block-Scaled Number Formats for Neural Networks [1.042733720689638]
  Quantization reduces the model's hardware costs, such as data movement, storage, and operations like multiply and addition. More exotic numerical encodings, such as block-scaled number formats, have shown advantages for utilizing a fixed bit budget to encode model parameters. This paper presents error diffusion (ED) for post-training quantization with support for block-scaled data formats.
  arXiv  Detail & Related papers  (2024-10-15T02:40:50Z)
• On the Embedding Collapse when Scaling up Recommendation Models [53.66285358088788]
  We identify the embedding collapse phenomenon as the inhibition of scalability, wherein the embedding matrix tends to occupy a low-dimensional subspace. We propose a simple yet effective multi-embedding design incorporating embedding-set-specific interaction modules to learn embedding sets with large diversity.
  arXiv  Detail & Related papers  (2023-10-06T17:50:38Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• MEIM: Multi-partition Embedding Interaction Beyond Block Term Format for Efficient and Expressive Link Prediction [3.718476964451589]
  We introduce the Multi- Partition Embedding Interaction iMproved beyond block term format (MEIM) model. MEIM improves expressiveness while still being highly efficient, helping it to outperform strong baselines and achieve state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-30T17:20:03Z)
• Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Models [57.933500846742234]
  Recent work recognizes that structured outliers are the critical bottleneck for quantization performance. We propose an outlier suppression framework including two components: Gamma Migration and Token-Wise Clipping. This framework effectively suppresses the outliers and can be used in a plug-and-play mode.
  arXiv  Detail & Related papers  (2022-09-27T12:05:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.