MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference

• URL: http://arxiv.org/abs/2604.00529v1
• Date: Wed, 01 Apr 2026 06:12:19 GMT
• Title: MF-QAT: Multi-Format Quantization-Aware Training for Elastic Inference
• Abstract summary: Quantization-aware training (QAT) is typically performed for a single target numeric format.<n>We study multi-format QAT, where a single model is trained to be robust across multiple quantization formats.<n>We propose the Slice-and-Scale conversion procedure for both MXINT and MXFP that converts a high-precision representation into lower-precision formats without re-training.
• Score: 1.938624051379935
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Quantization-aware training (QAT) is typically performed for a single target numeric format, while practical deployments often need to choose numerical precision at inference time based on hardware support or runtime constraints. We study multi-format QAT, where a single model is trained to be robust across multiple quantization formats. We find that multi-format QAT can match single-format QAT at each target precision, yielding one model that performs well overall across different formats, even formats that were not seen during training. To enable practical deployment, we propose the Slice-and-Scale conversion procedure for both MXINT and MXFP that converts a high-precision representation into lower-precision formats without re-training. Building on this, we introduce a pipeline that (i) trains a model with multi-format QAT, (ii) stores a single anchor format checkpoint (MXINT8/MXFP8), and (iii) allows on-the-fly conversion to lower MXINT or MXFP formats at runtime with negligible-or no-additional accuracy degradation. Together, these components provide a practical path to elastic precision scaling and allow selecting the runtime format at inference time across diverse deployment targets.

Related papers

• Improving Cross-Format Robustness in Language Models with Multi-Format Training [17.87388405562304]
  Cross-format robustness is the extent to which a model answers the same underlying question consistently across formats.<n>We compare full-format training with FormatMix, which expands only a subset of training items into multiple equivalent formats.<n>We find that expanding only about 30% of the training set into multiple formats often recovers most of the gain from full-format training.
  arXiv  Detail & Related papers  (2026-06-10T04:07:41Z)
• dMX: Differentiable Mixed-Precision Assignment for Low-Precision Floating-Point Formats [1.3357020021155022]
  dMX is a mixed-precision quantization framework for learnable floating-point bit-width assignment.<n>We study its application for the microscaling floating-point (MXFP) family of data types defined by the Open Compute Project standard.
  arXiv  Detail & Related papers  (2026-06-02T18:23:20Z)
• SQ-format: A Unified Sparse-Quantized Hardware-friendly Data Format for LLMs [8.787017031267482]
  Post-training quantization (PTQ) plays a crucial role in the democratization of large language models (LLMs)<n>Existing low-bit quantization and sparsification techniques are difficult to balance accuracy and efficiency due to the limited hardware support.<n>We propose the Sparse-Quantized Format (SQ-format), which is a unified data format for quantization and sparsification.
  arXiv  Detail & Related papers  (2025-12-05T03:58:04Z)
• CDLM: Consistency Diffusion Language Models For Faster Sampling [54.886467592798]
  Diffusion Language Models (DLMs) offer a promising parallel generation paradigm but suffer from slow inference.<n>We introduce CDLM, a training-based acceleration method that simultaneously tackles both bottlenecks.<n>Experiments show CDLM achieves 3.6x-14.5x lower latency while maintaining competitive accuracy on math and coding tasks.
  arXiv  Detail & Related papers  (2025-11-24T16:21:25Z)
• Optimal Formats for Weight Quantisation [6.073675653083644]
  We propose a framework for systematic design and analysis of quantisation formats.<n>We show that the strong practical performance of popular formats comes from their ability to represent values using variable-length codes.
  arXiv  Detail & Related papers  (2025-05-19T11:26:54Z)
• ML-SpecQD: Multi-Level Speculative Decoding with Quantized Drafts [79.62448915248926]
  Speculative decoding (SD) has emerged as a method to accelerate LLM inference without sacrificing accuracy over the 16-bit model inference.<n>We propose using MXFP4 models as drafts in a plug-and-play fashion since the MXFP4 Weight-Only-Quantization (WOQ) merely direct-casts the BF16 target model weights to MXFP4.<n>In practice, our plug-and-play solution gives speedups up to 2x over the BF16 baseline.
  arXiv  Detail & Related papers  (2025-03-17T08:38:45Z)
• Extreme Compression of Large Language Models via Additive Quantization [59.3122859349777]
  Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval. We provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed.
  arXiv  Detail & Related papers  (2024-01-11T18:54:44Z)
• Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formatting [68.19544657508509]
  Large language models (LLMs) are adopted as a fundamental component of language technologies. We find that several widely used open-source LLMs are extremely sensitive to subtle changes in prompt format in few-shot settings. We propose an algorithm that rapidly evaluates a sampled set of plausible prompt formats for a given task, and reports the interval of expected performance without accessing model weights.
  arXiv  Detail & Related papers  (2023-10-17T15:03:30Z)
• MatFormer: Nested Transformer for Elastic Inference [91.45687988953435]
  MatFormer is a novel Transformer architecture designed to provide elastic inference across diverse deployment constraints.<n>MatFormer achieves this by incorporating a nested Feed Forward Network (FFN) block structure within a standard Transformer model.<n>We show that a 850M decoder-only MatFormer language model (MatLM) allows us to extract multiple smaller models spanning from 582M to 850M parameters.
  arXiv  Detail & Related papers  (2023-10-11T17:57:14Z)
• Integer or Floating Point? New Outlooks for Low-Bit Quantization on Large Language Models [17.055400141733124]
  Low-bit integer formats (e.g., INT8/INT4) have been the conventional choice for large language models (LLMs) Low-bit floating-point formats (e.g., FP8/FP4) offer a compelling alternative and are gaining support from cutting-edge hardware, such as NVIDIA's H100 GPU. We propose the Mixture of Formats Quantization (MoFQ), which selects the optimal format on a layer-wise basis.
  arXiv  Detail & Related papers  (2023-05-21T05:28:37Z)
• Bilaterally Slimmable Transformer for Elastic and Efficient Visual Question Answering [75.86788916930377]
  bilaterally slimmable Transformer (BST) integrated into arbitrary Transformer-based VQA models. One slimmed MCAN-BST submodel achieves comparable accuracy on VQA-v2. Smallest MCAN-BST submodel has 9M parameters and 0.16G FLOPs during inference.
  arXiv  Detail & Related papers  (2022-03-24T02:26:04Z)
• Towards Efficient Post-training Quantization of Pre-trained Language Models [85.68317334241287]
  We study post-training quantization(PTQ) of PLMs, and propose module-wise quantization error minimization(MREM), an efficient solution to mitigate these issues. Experiments on GLUE and SQuAD benchmarks show that our proposed PTQ solution not only performs close to QAT, but also enjoys significant reductions in training time, memory overhead, and data consumption.
  arXiv  Detail & Related papers  (2021-09-30T12:50:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.