Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space

• URL: http://arxiv.org/abs/2608.30908v1
• Date: Mon, 31 Aug 2026 14:54:30 GMT
• Title: Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
• Abstract summary: Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form.<n>GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes.
• Score: 78.05046281165885
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning Low-bit models aims to adapt a quantized model while keeping the final deployed checkpoint in the same low-bit form. This setting is practically important as it reduces memory and inference cost for storage and deployment. Under this constraint, adaptation becomes an optimization problem over quantization codes and scales. Existing continuous low-bit training is efficient, but it can be distorted by straight through estimation error or by post-quantize gap; discrete search is deployment-faithful, but it is often too inefficient under a finite training budget. We propose code surrogate gradient as the first order signal in deployable code space to acceleate optimization, and performing guided search to preserve deployment faithfulness. Experiments across arithmetic reasoning, instruction following, and structured language understanding show that GradCodes consistently improves fine-tuning low-bit models across different quantization datatypes. Code is provided at https://github.com/ovo67/GradCodes.

Related papers

• Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications [0.9174244628582179]
  We propose a novel algorithm that modifies the gradients returned by training examples, specifically, scaling down the gradients of correctly classified examples using a dynamic scaler.<n>Experiments on a variety of benchmark datasets, spanning multiple tasks and large pretrained models, demonstrate that our method consistently reduces performance variance and surpasses the accuracy of existing approaches.
  arXiv  Detail & Related papers  (2026-04-30T15:16:00Z)
• Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression [57.54335545892155]
  We introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook.<n>Our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines.
  arXiv  Detail & Related papers  (2025-10-23T20:19:48Z)
• TruncQuant: Truncation-Ready Quantization for DNNs with Flexible Weight Bit Precision [8.532216260938478]
  Truncation is an effective approach for achieving lower bit precision mapping.<n>Current quantization-aware training schemes are not designed for the truncation process.<n>We propose TruncQuant, a novel truncation-ready training scheme allowing flexible bit precision through bit-shifting in runtime.
  arXiv  Detail & Related papers  (2025-06-13T03:08:18Z)
• Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics [64.62231094774211]
  Statefuls (e.g., Adam) maintain auxiliary information even 2x the model size in order to achieve optimal convergence.<n>SOLO enables Adam-styles to maintain quantized states with precision as low as 3 bits, or even 2 bits.<n>SOLO can thus be seamlessly applied to Adam-styles, leading to substantial memory savings with minimal accuracy loss.
  arXiv  Detail & Related papers  (2025-05-01T06:47:45Z)
• Sparse is Enough in Fine-tuning Pre-trained Large Language Models [98.46493578509039]
  We propose a gradient-based sparse fine-tuning algorithm, named Sparse Increment Fine-Tuning (SIFT) We validate its effectiveness on a range of tasks including the GLUE Benchmark and Instruction-tuning.
  arXiv  Detail & Related papers  (2023-12-19T06:06:30Z)
• Maestro: Uncovering Low-Rank Structures via Trainable Decomposition [15.254107731735553]
  Deep Neural Networks (DNNs) have been a large driver for AI breakthroughs in recent years. They have been getting increasingly large as they become more accurate and safe. This means that their training becomes increasingly costly and time-consuming. We propose Maestro, a framework for trainable low-rank layers.
  arXiv  Detail & Related papers  (2023-08-28T23:08:15Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.