Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free

• URL: http://arxiv.org/abs/2505.03810v2
• Date: Thu, 14 Aug 2025 07:02:58 GMT
• Title: Grouped Sequency-arranged Rotation: Optimizing Rotation Transformation for Quantization for Free
• Abstract summary: We introduce a novel, training-free approach to construct an improved rotation matrix, addressing the limitations of current methods.<n>Key contributions include leveraging the Walsh-Hadamard transform with sequency ordering, which clusters similar frequency components to reduce quantization error.<n>Our method demonstrates robust performance on reasoning tasks and Perplexity (PPL) score on WikiText-2.
• Score: 5.074812070492738
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Large Language Models (LLMs) face deployment challenges due to high computational costs, and while Post-Training Quantization (PTQ) offers a solution, existing rotation-based methods struggle at very low bit-widths like 2-bit. We introduce a novel, training-free approach to construct an improved rotation matrix, addressing the limitations of current methods. The key contributions include leveraging the Walsh-Hadamard transform with sequency ordering, which clusters similar frequency components to reduce quantization error compared to standard Hadamard matrices, significantly improving performance. Furthermore, we propose a Grouped Sequency-arranged Rotation (GSR) using block-diagonal matrices with smaller Walsh blocks, effectively isolating outlier impacts and achieving performance comparable to optimization-based methods without requiring any training. Our method demonstrates robust performance on reasoning tasks and Perplexity (PPL) score on WikiText-2. Our method also enhances results even when applied over existing learned rotation techniques.

Related papers

• ReSpinQuant: Efficient Layer-Wise LLM Quantization via Subspace Residual Rotation Approximation [32.6734400537211]
  Post-Training Quantization (PTQ) has emerged as a promising solution for mitigating activation outliers in the quantization of Large Language Models (LLMs)<n>Layer-wise transformation methods emerged, achieving superior accuracy through localized adaptation.<n>However, layer-wise methods cannot fuse activation rotation matrices into weights, requiring online computations and causing significant overhead.<n>We propose ReSpinQuant, a quantization framework that resolves such overhead by leveraging offline activation rotation fusion and matching basis.
  arXiv  Detail & Related papers  (2026-04-13T07:00:26Z)
• Inverse-Free Sparse Variational Gaussian Processes [24.916660624139638]
  We propose a drop-in replacement for sparse variational GP approximations.<n>We show that our method recovers similar performance to traditional methods, and can be faster than baselines when well tuned.
  arXiv  Detail & Related papers  (2026-04-01T09:53:04Z)
• WUSH: Near-Optimal Adaptive Transforms for LLM Quantization [52.77441224845925]
  Quantization to low bitwidth is a standard approach for deploying large language models.<n>A few extreme weights and activations stretch the dynamic range and reduce the effective resolution of the quantizer.<n>We derive, for the first time, closed-form optimal linear blockwise transforms for joint weight-activation quantization.
  arXiv  Detail & Related papers  (2025-11-30T16:17:34Z)
• Proximal Algorithm Unrolling: Flexible and Efficient Reconstruction Networks for Single-Pixel Imaging [45.39911367007956]
  Deep-unrolling and plug-and-play approaches have become the de-facto for single-pixel imaging (SPI) inverse problem.<n>In this paper, we address the challenge of integrating the strengths of both classes of solvers.
  arXiv  Detail & Related papers  (2025-05-29T07:16:57Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• Robust PCA Based on Adaptive Weighted Least Squares and Low-Rank Matrix Factorization [2.983818075226378]
  We propose a novel RPCA model that integrates adaptive weight factor update during initial component instability.<n>Our method outperforms existing non-inspired regularization approaches, offering superior performance and efficiency.
  arXiv  Detail & Related papers  (2024-12-19T08:31:42Z)
• Multiple Rotation Averaging with Constrained Reweighting Deep Matrix Factorization [22.487393413405954]
  Multiple rotation averaging plays a crucial role in computer vision and robotics domains. This paper proposes an effective rotation averaging method for mining data patterns in a learning manner.
  arXiv  Detail & Related papers  (2024-09-15T16:50:27Z)
• DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMs [40.48697728884967]
  Quantization of large language models (LLMs) faces significant challenges, particularly due to the presence of outlier activations. Traditional approaches predominantly address Normal Outliers, which are activations across all tokens with relatively large magnitudes. We introduce DuQuant, a novel approach that utilizes rotation and permutation transformations to more effectively mitigate both massive and normal outliers.
  arXiv  Detail & Related papers  (2024-06-03T18:27:44Z)
• Variable Substitution and Bilinear Programming for Aligning Partially Overlapping Point Sets [48.1015832267945]
  This research presents a method to meet requirements through the minimization objective function of the RPM algorithm. A branch-and-bound (BnB) algorithm is devised, which solely branches over the parameters, thereby boosting convergence rate. Empirical evaluations demonstrate better robustness of the proposed methodology against non-rigid deformation, positional noise, and outliers, when compared with prevailing state-of-the-art transformations.
  arXiv  Detail & Related papers  (2024-05-14T13:28:57Z)
• An Efficient Algorithm for Clustered Multi-Task Compressive Sensing [60.70532293880842]
  Clustered multi-task compressive sensing is a hierarchical model that solves multiple compressive sensing tasks. The existing inference algorithm for this model is computationally expensive and does not scale well in high dimensions. We propose a new algorithm that substantially accelerates model inference by avoiding the need to explicitly compute these covariance matrices.
  arXiv  Detail & Related papers  (2023-09-30T15:57:14Z)
• Givens Coordinate Descent Methods for Rotation Matrix Learning in Trainable Embedding Indexes [19.716527782586788]
  We propose a family of block Givens coordinate descent algorithms to learn rotation matrix. Compared to the state-of-the-art SVD method, the Givens algorithms are much more parallelizable.
  arXiv  Detail & Related papers  (2022-03-09T22:58:56Z)
• ART-Point: Improving Rotation Robustness of Point Cloud Classifiers via Adversarial Rotation [89.47574181669903]
  In this study, we show that the rotation robustness of point cloud classifiers can also be acquired via adversarial training. Specifically, our proposed framework named ART-Point regards the rotation of the point cloud as an attack. We propose a fast one-step optimization to efficiently reach the final robust model.
  arXiv  Detail & Related papers  (2022-03-08T07:20:16Z)
• A Reinforcement Learning Approach for Sequential Spatial Transformer Networks [6.585049648605185]
  We formulate the task as a Markovian Decision Process (MDP) and use RL to solve this sequential decision-making problem. In our method, we are not bound to the differentiability of the sampling modules. We design multiple experiments to verify the effectiveness of our method using cluttered MNIST and Fashion-MNIST datasets.
  arXiv  Detail & Related papers  (2021-06-27T17:41:17Z)
• Hybrid Trilinear and Bilinear Programming for Aligning Partially Overlapping Point Sets [85.71360365315128]
  In many applications, we need algorithms which can align partially overlapping point sets are invariant to the corresponding corresponding RPM algorithm. We first show that the objective is a cubic bound function. We then utilize the convex envelopes of trilinear and bilinear monomial transformations to derive its lower bound. We next develop a branch-and-bound (BnB) algorithm which only branches over the transformation variables and runs efficiently.
  arXiv  Detail & Related papers  (2021-01-19T04:24:23Z)
• Effective Dimension Adaptive Sketching Methods for Faster Regularized Least-Squares Optimization [56.05635751529922]
  We propose a new randomized algorithm for solving L2-regularized least-squares problems based on sketching. We consider two of the most popular random embeddings, namely, Gaussian embeddings and the Subsampled Randomized Hadamard Transform (SRHT)
  arXiv  Detail & Related papers  (2020-06-10T15:00:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.