Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training

• URL: http://arxiv.org/abs/2605.26189v2
• Date: Wed, 27 May 2026 01:07:53 GMT
• Title: Max-Window Scale Estimation for Near-Lossless HiF8 W8A8 Quantization-Aware Training
• Abstract summary: Quantization-aware training (QAT) with low-bit floating-point formats enables efficient LLM deployment, yet introduces subtle failure modes invisible to standard training metrics.<n>We present a systematic study of HiF8 W8A8 QAT for OpenPangu-Embedded-1B through the lens of Delayed Scaling (DTS)<n>We identify and disentangle two failure modes: (i)amax saturation, where delayed scale estimates corrupt knowledge-sensitive representations via forward-pass clipping, and (ii)catastrophic forgetting, where an aggressive learning rate overwrites pretrained commonsense knowledge independently of quantization
• Score: 5.976336341654394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantization-aware training (QAT) with low-bit floating-point formats enables efficient LLM deployment, yet introduces subtle failure modes invisible to standard training metrics. We present a systematic study of HiF8 W8A8 QAT for OpenPangu-Embedded-1B through the lens of Delayed Tensor Scaling (DTS). Across eight controlled experiments, we identify and disentangle two orthogonal failure modes: (i)amax saturation, where delayed scale estimates silently corrupt knowledge-sensitive representations via forward-pass clipping, and (ii)catastrophic forgetting, where an aggressive learning rate overwrites pretrained commonsense knowledge independently of quantization. Neither is detectable from training loss alone. We address amax saturation with a conservative max-algorithm DTS strategy over a 64-step history window, and mitigate forgetting via a 500-step BF16 warmup followed by QAT at lr=10^{-5}. Both fixes are necessary and sufficient: our final configuration achieves 0.43% MMLU drop, 0.58% HellaSwag drop, and 0.22% ARC-Challenge drop versus a matched BF16 baseline, with a training loss APE of only 0.11% over 10,000 steps.

Related papers

• Solution for UCF UrbanTwin LUMPI Track: Sim-to-Real Urban LiDAR 3D Object Detection [54.03436243843709]
  We present our solution to the LUMPI track of the UCF UrbanTwin Sim2Real LiDAR Challenge at the 6th DriveX Workshop, ECCV 2026.<n>The detector must be trained only on synthetic data and is evaluated on 50 held-out real LiDAR frames.<n>A separate 50-frame synthetic submission is evaluated for point-cloud realism.
  arXiv  Detail & Related papers  (2026-09-07T15:03:36Z)
• SS-ESOAP: Self-Scaled Adaptive Preconditioning for Physics-Informed Learning [70.71795707610961]
  method augments SOAP-style preconditioning with a scalar secant-energy correction adapted to Kronecker geometry.<n>Method attains the lowest final residual on six of eight PDE benchmarks.<n>Three-seed $L2$ and $H1$ errors on four representative PDEs support the link between lower residuals and improved solution accuracy.
  arXiv  Detail & Related papers  (2026-08-29T21:34:47Z)
• Ternary Mamba: Grouped Quantization-Aware Training of W1.58A16 State Space Models [0.0]
  State Space Models (SSMs) such as Mamba-2 offer linear-time inference but their memory footprint limits edge deployment.<n>We show a pretrained checkpoint suffices, reducing the marginal token budget by 1,000x.<n>We achieve 48.1% zero-shot accuracy in just 102M tokens (4 GPU-hours, single H100)
  arXiv  Detail & Related papers  (2026-06-16T16:18:21Z)
• Fine-Tuning Vision-Language Models for Understanding Current Damage and Scoring Priority with Quality Guard Agent [0.0]
  Bridge inspection in Japan requires mandatory visual assessments every five years.<n> qualitative damage ratings (levels a-e) assigned by different engineers exhibit significant inter-rater variability.<n>This paper presents a methodology for automating bridge damage understanding and repair priority scoring.
  arXiv  Detail & Related papers  (2026-05-24T21:11:50Z)
• AGoQ: Activation and Gradient Quantization for Memory-Efficient Distributed Training of LLMs [9.35503487039164]
  Quantization is a key method for reducing the GPU memory requirement of training large language models (LLMs)<n>We introduce AGoQ, incorporating two new techniques: 1) a layer-aware activation quantization algorithm that allocates appropriate bit-widths for activations of various layers based on their types and pipeline stages to achieve near 4-bit activation storage, and 2) a gradient quantization algorithm that reduces memory usage and shortens communication time by employing 8-bit gradient storage and precision-preserving 8-bit All-Reduce communication.
  arXiv  Detail & Related papers  (2026-05-01T09:39:03Z)
• Facial Emotion Recognition on FER-2013 using an EfficientNetB2-Based Approach [0.0]
  Detection of human emotions based on facial images in real-world scenarios is a difficult task due to low image quality, variations in lighting, pose changes, background distractions, small inter-class variations, noisy crowd-sourced labels, and severe class imbalance.<n>We address these challenges using a lightweight and efficient facial emotion recognition pipeline based on EfficientNetB2.<n>The model is trained using a stratified 87.5%/12.5% train-validation split while keeping the official test set intact, achieving a test accuracy of 68.78% with nearly ten times fewer parameters than VGG16-based baselines.
  arXiv  Detail & Related papers  (2026-01-26T07:29:50Z)
• Pretraining Large Language Models with NVFP4 [53.235038214986865]
  We introduce a novel approach for stable and accurate training of large language models (LLMs) using the NVFP4 format.<n>Our method integrates two-dimensional quantization scheme for consistent representations across both the forward and backward passes.<n>Our results show that the model trained with our NVFP4-based pretraining technique achieves training loss and downstream task accuracies comparable to an FP8 baseline.
  arXiv  Detail & Related papers  (2025-09-29T17:53:17Z)
• Stable-SPAM: How to Train in 4-Bit More Stably than 16-Bit Adam [94.00189300897694]
  Low-bit precision amplifies sensitivity learning rates and often causes unstable gradient norms.<n>We propose Stable-SPAM, which incorporates enhanced gradient normalization and clipping techniques.<n>Experiments show that Stable-SPAM effectively stabilizes gradient norms in 4-bit training, delivering superior performance compared to Adam and SPAM.
  arXiv  Detail & Related papers  (2025-02-24T11:09:15Z)
• Continual Quantization-Aware Pre-Training: When to transition from 16-bit to 1.58-bit pre-training for BitNet language models? [5.67099529296254]
  Large language models (LLMs) require immense resources for training and inference.<n>Recent research suggests that training LLMs with 1.58 bits per weight parameter from scratch can maintain model accuracy.
  arXiv  Detail & Related papers  (2025-02-17T15:21:11Z)
• Stable and low-precision training for large-scale vision-language models [108.62077651227607]
  We introduce new methods for accelerating and stabilizing training for large language-vision models. For acceleration, we introduce SwitchBack, a linear layer for int8 quantized training which provides a speed-up of 13-25%. For stability, we analyze loss spikes and find they consistently occur 1-8 after the squared gradients become under-estimated.
  arXiv  Detail & Related papers  (2023-04-25T17:38:18Z)
• Do We Need Zero Training Loss After Achieving Zero Training Error? [76.44358201918156]
  We propose a direct solution called emphflooding that intentionally prevents further reduction of the training loss when it reaches a reasonably small value. We experimentally show that flooding improves performance and, as a byproduct, induces a double descent curve of the test loss.
  arXiv  Detail & Related papers  (2020-02-20T12:50:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.