Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay

• URL: http://arxiv.org/abs/2605.01330v1
• Date: Sat, 02 May 2026 08:49:51 GMT
• Title: Colinearity Decay: Training Quantization-Friendly ViTs with Outlier Decay
• Abstract summary: Colinearity-Decay (CD) is a structural regularizer for ordered matrix pairs within Transformer blocks.<n>CD penalizes detrimental cross-matrix alignment and mitigates extreme activations without altering the architecture or task loss.
• Score: 9.836950534671775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-bit quantization is a practical route for efficiently deploying vision Transformers, yet activation outliers complicate fully quantized deployment. Existing methods either handle quantization post-training or suppress large activations during training; however, aggressively restricting outliers in vision models can lead to a poorer trade-off between full-precision and quantized accuracy. We argue that rather than simply suppressing outliers, the training objective should control the structural amplification that makes them harmful. To this end, we introduce Colinearity-Decay (CD), a structural regularizer for ordered matrix pairs within Transformer blocks. CD penalizes detrimental cross-matrix alignment and mitigates extreme activations without altering the architecture or task loss. Applied as a decoupled update, CD is non-invasive and introduces minimal training overhead. Across ImageNet-1K pre-training, COCO detection, and downstream fine-tuning, CD consistently boosts quantized accuracy across multiple pipelines while preserving, or even improving, full-precision performance. Ultimately, our results demonstrate that structural regularization effectively prepares vision Transformers for low-bit deployment with zero inference-time overhead.

Related papers

• Dense Structural Compression of Transformers via Gauge-Correct Channel Removal [11.681900731333926]
  Inference energy per token drives the cost and carbon footprint of deployed transformers.<n>We develop a methodology to adapt structural complexity during training to maximize utility per unit compute.<n>Channel penalties drive entire tensor slices to zero to enable physical removal while preserving density and the network function.
  arXiv  Detail & Related papers  (2026-09-07T09:20:59Z)
• SCULPT: Training Edge Vision Models for Post-Training Quantization Readiness [0.0]
  Edge vision models are difficult to deploy on resource-constrained hardware.<n>We present SCULPT, a training-time method that improves PTQ readiness during ordinary FP32 fine-tuning.
  arXiv  Detail & Related papers  (2026-09-01T18:10:19Z)
• Taming Curvature: Architecture Warm-Up for Stable Transformer Training [52.6838951211596]
  We introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power with Hessian-vector products.<n>We find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth.<n> Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities.
  arXiv  Detail & Related papers  (2026-06-15T14:16:56Z)
• Self-supervised pretraining for an iterative image size agnostic vision transformer [60.09702846704075]
  Vision Transformers (ViTs) dominate self-supervised learning (SSL)<n>We introduce a novel sequential-to-global SSL framework based on DINO's self-distillation objective.<n>We achieve competitive performance on ImageNet-1K and downstream classification tasks.
  arXiv  Detail & Related papers  (2026-04-22T09:53:28Z)
• Efficient Multi-bit Quantization Network Training via Weight Bias Correction and Bit-wise Coreset Sampling [19.052294458935595]
  Multi-bit quantization networks enable flexible deployment of deep neural networks by supporting multiple precision levels within a single model.<n>Existing approaches suffer from significant training overhead as full-dataset updates are repeated for each supported bit-width.<n>We propose two techniques that greatly reduce the training overhead without compromising model utility.
  arXiv  Detail & Related papers  (2025-10-23T15:49:02Z)
• PT$^2$-LLM: Post-Training Ternarization for Large Language Models [52.4629647715623]
  Large Language Models (LLMs) have shown impressive capabilities across diverse tasks, but their large memory and compute demands hinder deployment.<n>We propose PT$2$-LLM, a post-training ternarization framework tailored for LLMs.<n>At its core is an Asymmetric Ternary Quantizer equipped with a two-stage refinement pipeline.
  arXiv  Detail & Related papers  (2025-09-27T03:01:48Z)
• Boosted Training of Lightweight Early Exits for Optimizing CNN Image Classification Inference [47.027290803102666]
  We introduce a sequential training approach that aligns branch training with inference-time data distributions.<n>Experiments on the CINIC-10 dataset with a ResNet18 backbone demonstrate that BTS-EE consistently outperforms non-boosted training.<n>These results offer practical efficiency gains for applications such as industrial inspection, embedded vision, and UAV-based monitoring.
  arXiv  Detail & Related papers  (2025-09-10T06:47:49Z)
• Progressive Element-wise Gradient Estimation for Neural Network Quantization [2.1413624861650358]
  Quantization-Aware Training (QAT) methods rely on the Straight-Through Estimator (STE) to address the non-differentiability of discretization functions.<n>We propose Progressive Element-wise Gradient Estimation (PEGE) to address discretization errors between continuous and quantized values.<n>PEGE consistently outperforms existing backpropagation methods and enables low-precision models to match or even outperform the accuracy of their full-precision counterparts.
  arXiv  Detail & Related papers  (2025-08-27T15:59:36Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• Efficient local linearity regularization to overcome catastrophic overfitting [59.463867084204566]
  Catastrophic overfitting (CO) in single-step adversarial training results in abrupt drops in the adversarial test accuracy (even down to 0%) We introduce a regularization term, called ELLE, to mitigate CO effectively and efficiently in classical AT evaluations.
  arXiv  Detail & Related papers  (2024-01-21T22:55:26Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.