Adaptive Block-Scaled Data Types

• URL: http://arxiv.org/abs/2603.28765v1
• Date: Mon, 30 Mar 2026 17:59:33 GMT
• Title: Adaptive Block-Scaled Data Types
• Abstract summary: NVFP4 suffers from its error distribution, resulting in large amounts of quantization error on near-maximal values.<n>We design new Adaptive Block-Scaled Data Types that can adapt to the distribution of their input values.<n> IF4 outperforms existing 4-bit block-scaled formats, achieving lower loss during quantized training and achieving higher accuracy on many tasks in post-training quantization.
• Score: 7.634741713346681
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: NVFP4 has grown increasingly popular as a 4-bit format for quantizing large language models due to its hardware support and its ability to retain useful information with relatively few bits per parameter. However, the format is not without limitations: recent work has shown that NVFP4 suffers from its error distribution, resulting in large amounts of quantization error on near-maximal values in each group of 16 values. In this work, we leverage this insight to design new Adaptive Block-Scaled Data Types that can adapt to the distribution of their input values. For four-bit quantization, our proposed IF4 (Int/Float 4) data type selects between FP4 and INT4 representations for each group of 16 values, which are then scaled by an E4M3 scale factor as is done with NVFP4. The selected data type is denoted using the scale factor's sign bit, which is currently unused in NVFP4, and we apply the same insight to design formats for other bit-widths, including IF3 and IF6. When used to quantize language models, we find that IF4 outperforms existing 4-bit block-scaled formats, achieving lower loss during quantized training and achieving higher accuracy on many tasks in post-training quantization. We additionally design and evaluate an IF4 Multiply-Accumulate (MAC) unit to demonstrate that IF4 can be implemented efficiently in next-generation hardware accelerators. Our code is available at https://github.com/mit-han-lab/fouroversix.

Related papers

• FOCUS: FP4 Optimization via Coupled-Relaxation and Dual-Granularity Scaling [23.78963716646984]
  FOCUS is a post-training quantization framework with end-to-end scale learning for FP4 optimization.<n>It relaxes the tight coupling between quantization and dequantization scales with a learnable full-precision coefficient.<n>Experiments show that FOCUS achieves state-of-the-art FP4 accuracy under both MXFP4 and NVFP4 formats.
  arXiv  Detail & Related papers  (2026-08-03T07:54:56Z)
• Grid Games: The Power of Multiple Grids for Quantizing Large Language Models [50.885349461958384]
  A major recent advance in quantization is given by microscaled 4-bit formats such as NVFP4 and MXFP4, quantizing values into small groups sharing a scale.<n>We formalize the power-of-two-grids (PO2) problem, and provide theoretical results showing that practical small-group formats such as MXFP or NVFP can benefit significantly from PO2 grids.<n>Results for post-training quantization of standard open models and pre-training of Llama-like models show that adaptive grids consistently improve accuracy vs single-grid FP4 under both weight-only and weight+activation.
  arXiv  Detail & Related papers  (2026-05-12T16:09:02Z)
• HiFloat4 Format for Language Model Inference [25.863121704892734]
  This paper introduces HiFloat4 (HiF4), a block floating-point data format tailored for deep learning.<n>Each HiF4 unit packs 64 4-bit elements with 32 bits of shared scaling metadata, averaging 4.5 bits per value.<n>Results show that HiF4 achieves higher average accuracy than the state-of-the-art NVFP4 format across multiple models and diverse downstream tasks.
  arXiv  Detail & Related papers  (2026-02-11T19:07:36Z)
• Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling [13.357423392911036]
  We introduce Four Over Six (4/6), a modification to the NVFP4 quantization algorithm that evaluates two potential scale factors for each block of values.<n>We find that for some blocks, scaling to smaller FP4 values makes the distribution of representable values more uniform.<n>We also find that 4/6 can be easily incorporated into many different post-training quantization methods and generally improves downstream accuracy.
  arXiv  Detail & Related papers  (2025-12-01T18:59:45Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization [77.67818998672516]
  We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization.<n>We introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm.<n>We show that MR-GPTQ matches or outperforms state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-09-27T09:22:21Z)
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models [57.04178959678024]
  We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit. We provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x.
  arXiv  Detail & Related papers  (2023-10-13T17:15:05Z)
• Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models [17.055400141733124]
  Low-bit integer formats (e.g., INT8/INT4) have been the conventional choice for large language models (LLMs) Low-bit floating-point formats (e.g., FP8/FP4) offer a compelling alternative and are gaining support from cutting-edge hardware, such as NVIDIA's H100 GPU. We propose the Mixture of Formats Quantization (MoFQ), which selects the optimal format on a layer-wise basis.
  arXiv  Detail & Related papers  (2023-05-21T05:28:37Z)
• The case for 4-bit precision: k-bit Inference Scaling Laws [75.4335600212427]
  Quantization methods reduce the number of bits required to represent each parameter in a model. The final model size depends on both the number of parameters of the original model and the rate of compression. We run more than 35,000 zero-shot experiments with 16-bit inputs and k-bit parameters to examine which quantization methods improve scaling for 3 to 8-bit precision.
  arXiv  Detail & Related papers  (2022-12-19T18:48:33Z)
• FP8 Formats for Deep Learning [49.54015320992368]
  We propose an 8-bit floating point (FP8) binary interchange format consisting of two encodings. E4M3's dynamic range is extended by not representing infinities and having only one mantissa bit-pattern for NaNs. We demonstrate the efficacy of the FP8 format on a variety of image and language tasks, effectively matching the result quality achieved by 16-bit training sessions.
  arXiv  Detail & Related papers  (2022-09-12T17:39:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.