AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation

• URL: http://arxiv.org/abs/2604.02525v1
• Date: Thu, 02 Apr 2026 21:24:15 GMT
• Title: AdaHOP: Fast and Accurate Low-Precision Training via Outlier-Pattern-Aware Rotation
• Abstract summary: Low-precision training employs Hadamard transforms to suppress outliers and mitigate quantization error.<n>Prior methods apply a fixed transform uniformly, despite substantial variation in outlier structures across tensors.<n>We show that this strategy is fundamentally flawed and propose AdaHOP.
• Score: 11.070767185562817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-precision training (LPT) commonly employs Hadamard transforms to suppress outliers and mitigate quantization error in large language models (LLMs). However, prior methods apply a fixed transform uniformly, despite substantial variation in outlier structures across tensors. Through the first systematic study of outlier patterns across weights, activations, and gradients of LLMs, we show that this strategy is fundamentally flawed: the effectiveness of Hadamard-based suppression depends on how the transform's smoothing direction aligns with the outlier structure of each operand -- a property that varies substantially across layers and computation paths. We characterize these patterns into three types: Row-wise, Column-wise, and None. Each pair requires a tailored transform direction or outlier handling strategy to minimize quantization error. Based on this insight, we propose AdaHOP (Adaptive Hadamard transform with Outlier-Pattern-aware strategy), which assigns each matrix multiplication its optimal strategy: Inner Hadamard Transform (IHT) where inner-dimension smoothing is effective, or IHT combined with selective Outlier Extraction (OE) -- routing dominant outliers to a high-precision path -- where it is not. Combined with hardware-aware Triton kernels, AdaHOP achieves BF16 training quality at MXFP4 precision while delivering up to 3.6X memory compression and 1.8X kernel acceleration} over BF16 full-precision training.

Related papers

• Low-Rank Ternary Adaptation for Fine-Tuning Transformers [54.3230438745856]
  Ternary transformers offer extreme memory and compute efficiency.<n>Existing low-bit LoRA-based methods cannot directly fine-tune ternary weights.<n>We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights.
  arXiv  Detail & Related papers  (2026-08-25T12:15:56Z)
• CausalGate: Causal Importance Distillation for Transformer Module Pruning [0.0]
  CausalGate is an intervention-guided framework for compute-efficient runtime inference.<n>It measures semantic damage via the Kullback-Leibler divergence of the final logit distribution.<n>It consistently outperforms prominent dynamic routing and layer-skipping baselines.
  arXiv  Detail & Related papers  (2026-07-21T20:19:59Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Towards Generalized Image Manipulation Localization via Score-based Model [38.03898206912958]
  We propose DiffIML, a novel framework that introduces score-based generative modeling to IML.<n>Under this formulation, diffusion models serve as an effective numerical solver for the learned score function.<n>Experiments on eight non-generative and three generative benchmarks demonstrate that DiffIML consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-16T08:39:54Z)
• Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation [0.5266471639657487]
  Vision Transformers (ViTs) achieve state-of-the-art performance on challenging vision tasks, but their deployment on edge devices is hindered.<n>Recent methods attempt to bypass this by replacing normalization layers with hardware-friendly scalar approximations.<n>We propose a highly efficient framework that utilizes genetic programming (GP) to evolve heterogeneous, layer-specific scalar functions.
  arXiv  Detail & Related papers  (2026-05-13T19:08:55Z)
• Tail-Aware Post-Training Quantization for 3D Geometry Models [58.79500829118265]
  Post-Training Quantization (PTQ) enables efficient inference without retraining.<n>PTQ fails to transfer effectively to 3D models due to intricate feature distributions and prohibitive calibration overhead.<n>We propose TAPTQ, a Tail-Aware Post-Training Quantization pipeline for 3D geometric learning.
  arXiv  Detail & Related papers  (2026-02-02T07:21:15Z)
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts [0.0]
  Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token.<n>Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks.<n>This paper introducesMoE (MLP-Experts), a training-free transformation that restructures the dense in transformer blocks into a static, high-cardinality mixture of experts.
  arXiv  Detail & Related papers  (2025-11-26T06:14:26Z)
• Adaptive Layer-Wise Transformations for Post-Training Quantization of Large Language Models [47.54958360970588]
  Large language models require significant computational resources for deployment.<n>Main obstacle to effective quantization lies in systematic outliers in activations and weights.<n>We propose an adaptive transformation selection framework that systematically determines optimal transformations on a per-layer basis.
  arXiv  Detail & Related papers  (2025-11-21T22:01:58Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [80.62659379624867]
  We propose Flow-GRPO, the first method to integrate online policy reinforcement learning into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Differential Equation into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original number of inference steps.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Micro-splatting: Multistage Isotropy-informed Covariance Regularization Optimization for High-Fidelity 3D Gaussian Splatting [1.5582756275568836]
  Micro-Splatting is a unified, in-training pipeline that preserves visual detail while drastically reducing model complexity.<n>On four object-centric benchmarks, Micro-Splatting reduces splat count and model size by up to 60% and shortens training by 20%.<n>Results demonstrate that Micro-Splatting delivers both compactness and high fidelity in a single, efficient, end-to-end framework.
  arXiv  Detail & Related papers  (2025-04-08T07:15:58Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts.<n>Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• MARS: Unleashing the Power of Variance Reduction for Training Large Models [56.67982828148859]
  We propose a unified training framework for deep neural networks.<n>We introduce three instances of MARS that leverage preconditioned gradient optimization.<n>Results indicate that the implementation of MARS consistently outperforms Adam.
  arXiv  Detail & Related papers  (2024-11-15T18:57:39Z)
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs [40.48697728884967]
  Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. We introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers.
  arXiv  Detail & Related papers  (2024-06-03T18:27:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.