MXNorm: Reusing MXFP block scales for efficient tensor normalisation

• URL: http://arxiv.org/abs/2603.13180v1
• Date: Fri, 13 Mar 2026 17:14:06 GMT
• Title: MXNorm: Reusing MXFP block scales for efficient tensor normalisation
• Abstract summary: We propose MXNorm, a drop-in replacement for RMSNorm that estimates the RMS using only the block scales calculated as part of the MXFP8 cast.<n>We show practical kernel speedups using only torch.compile of up to 2.4x for MXNorm over RMSNorm, corresponding to a 1.3% speedup in Llama 3 8B transformer layers in MXFP8 and a 2.6% speedup in NVFP4.
• Score: 39.74786083127591
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Matrix multiplication performance has long been the major bottleneck to scaling deep learning workloads, which has stimulated the design of new accelerators that use increasingly low-precision number formats. However, improvements in matrix multiplication performance have far outstripped improvements in performance on reductions and elementwise computations, which are still being performed in higher precision. In this work, we propose MXNorm, a drop-in replacement for RMSNorm that estimates the RMS using only the block scales calculated as part of the MXFP8 cast and enables a 32x decrease in the size of reduction needed for normalization. We validate our approximation method on pre-training of Llama 3 models of 125M, 1B and 8B parameters, finding minimal loss of training accuracy compared to a baseline using RMSNorm with MXFP8 matmuls. We also show practical kernel speedups using only torch.compile of up to 2.4x for MXNorm over RMSNorm, corresponding to a 1.3% speedup in Llama 3 8B transformer layers in MXFP8 and a 2.6% speedup in NVFP4.

Related papers

• MXSens: Sensitivity-Aware Mixed-Precision Quantization for Efficient LLM Inference [36.01709922688871]
  We introduce MXSens, a training-free method that assigns mixed mantissa bitwidths based on column- and layer-wise sensitivity.<n> MXSens achieves perplexities of 3.77 and 7.63 on LLaMA-2-70B and LLaMA-3-8B, respectively, substantially improving over existing baselines on WikiText-2.
  arXiv  Detail & Related papers  (2026-07-20T09:23:10Z)
• MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation [4.769740034530933]
  We propose a tile-based block design to increase hardware efficiency by reducing the burden of re-quantization process during the training with the MXSF format.<n>We present a training-inference accelerator that supports the MXSF format and it achieves similar accuracy to the BF16 baseline.
  arXiv  Detail & Related papers  (2026-05-23T04:21:57Z)
• Training Large Reasoning Models Efficiently via Progressive Thought Encoding [63.254758972725654]
  Large reasoning models (LRMs) excel on complex problems but face a critical barrier to efficiency.<n>We introduce Progressive Thought, a parameter-efficient fine-tuning method that enables LRMs to reason effectively under fixed-size caches.
  arXiv  Detail & Related papers  (2026-02-18T20:03:38Z)
• Unveiling the Potential of Quantization with MXFP4: Strategies for Quantization Error Reduction [1.8599887993390827]
  Open Compute Project (OCP) Microscaling standard MXFP4 lags behind NVIDIA's NVFP4 in accuracy.<n>We introduce two software-only techniques, Overflow-Aware Scaling (OAS) and Macro Block Scaling (MBS)<n>OAS and MBS reduce the end-to-end accuracy gap between MXFP4 and NVFP4 from about 10% to below 1% on average.
  arXiv  Detail & Related papers  (2026-01-30T23:24:17Z)
• Precision-Scalable Microscaling Datapaths with Optimized Reduction Tree for Efficient NPU Integration [3.146407420010677]
  We propose a hybrid precision-scalable reduction tree for MX MACs that combines the benefits of both approaches.<n>We integrate an 8x8 array of these MACs into the state-of-the-art (SotA) NPU integration platform, SNAX, to provide efficient control and data transfer.<n>Our integrated system achieves an energy efficiency of 657, 1438-1675, and 4065 GOPS/W, respectively, for MXINT8, MXFP8/6, and MXFP4, with a throughput of 64, 256, and 512 GOPS.
  arXiv  Detail & Related papers  (2025-11-09T10:24:17Z)
• INT v.s. FP: A Comprehensive Study of Fine-Grained Low-bit Quantization Formats [51.72056104795248]
  Modern AI hardware, such as Nvidia's Blackwell architecture, is increasingly embracing low-precision floating-point (FP) formats.<n>This paper systematically investigates the trade-offs between FP and integer (INT) formats.<n>We reveal a critical performance crossover: while FP excels in coarse-grained quantization, the comparison at fine-grained (block-wise) levels is more nuanced.
  arXiv  Detail & Related papers  (2025-10-29T15:11:53Z)
• Bridging the Gap Between Promise and Performance for Microscaling FP4 Quantization [77.67818998672516]
  We present the first comprehensive study of MXFP4 and NVFP4 for post-training quantization.<n>We introduce Micro-Rotated-GPTQ (MR-GPTQ), a variant of the classic GPTQ quantization algorithm.<n>We show that MR-GPTQ matches or outperforms state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2025-09-27T09:22:21Z)
• Orthogonal Finetuning Made Scalable [92.34573849209238]
  Orthogonal finetuning (OFT) offers highly parameter-efficient adaptation while preventing catastrophic forgetting, but its high runtime and memory demands limit practical deployment.<n>We identify the core computational bottleneck in OFT as its weight-centric implementation, which relies on costly matrix-matrix multiplications with cubic complexity.<n>We propose OFTv2, an input-centric reformulation that instead uses matrix-vector multiplications (i.e., matrix-free computation), reducing the computational cost to quadratic.<n>These modifications allow OFTv2 to achieve up to 10x faster training and 3x lower GPU memory usage without
  arXiv  Detail & Related papers  (2025-06-24T17:59:49Z)
• Training LLMs with MXFP4 [16.524414449291488]
  We present the first near-lossless training recipe that uses MXFP4 GEMMs, which are $2times$ faster than FP8 on supported hardware.<n>Our recipe computes $>1/2$ the training FLOPs in MXFP4, enabling an estimated speedup of $>1.3times$ over FP8 and $>1.7times$ over BF16 during backpropagation.
  arXiv  Detail & Related papers  (2025-02-27T23:01:31Z)
• AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference [6.699442219974261]
  AMXFP4 is a 4-bit asymmetric FP format that handles both issues using asymmetric shared scales.<n>AMXFP4 outperforms MXFP4 by 3% on VQA and exceeds rotation-based methods by 1.6% on CSQA.
  arXiv  Detail & Related papers  (2024-11-15T03:11:19Z)
• 8-bit Optimizers via Block-wise Quantization [57.25800395197516]
  Statefuls maintain statistics over time, e.g., the exponentially smoothed sum (SGD with momentum) or squared sum (Adam) of past values. This state can be used to accelerate optimization compared to plain gradient descent but uses memory that might otherwise be allocated to model parameters. In this paper, we develop first gradients that use 8-bit statistics while maintaining the performance levels of using 32-bit gradient states.
  arXiv  Detail & Related papers  (2021-10-06T15:43:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.