Fine-Tuning Low-Bit Models with Gradient in Quantized Code Space
Abstract Overview
This paper studies strict fully low-bit fine-tuning, where the adapted model must remain in the same quantized format at deployment, so optimization is performed over quantization codes and scales rather than a continuous weight space. The authors propose GradCodeS, which constructs a geometry-aware code surrogate gradient and uses it to guide a discrete guide-sample-evaluate-select search over valid low-bit candidates. The method separates proposal construction from update acceptance: gradients shape where to search, but the accepted state is chosen only by the realized loss of deployable low-bit checkpoints. Experiments cover arithmetic reasoning, instruction following, and structured semantic parsing on Qwen3-0.6B and Llama-3.2 1B/3B models, with evaluations across multiple 4-bit baselines and quantization datatypes.
Novelty
The main novelty is a first-order optimization signal defined directly in quantized code space, rather than reusing ordinary weight gradients or relying purely on zeroth-order search. The paper also combines this code-space surrogate gradient with deployment-faithful discrete candidate selection, and provides local theoretical results linking the surrogate to steepest descent and one-step code updates.
Results
Across the reported tasks and backbones, GradCodeS is the strongest fully 4-bit method, consistently improving over direct 4-bit quantization and other fully low-bit baselines such as PV-Tuning, QuZO, QZO, and QES. The method also shows robustness across NF4, INT4, and MXFP4, and its guided sampler yields higher candidate selection probability under the same budget than Gaussian, one-hop, or ordinary weight-gradient-guided sampling.
Key Points
- GradCodeS targets deployment-faithful low-bit fine-tuning by optimizing directly over quantized codes and scales, avoiding a mismatch between the trained state and the deployed checkpoint.
- Its core technical idea is a code surrogate gradient that accounts for quantization geometry, including codebook gaps and group scales, and then uses that signal to guide discrete candidate search.
- Empirically, the method delivers the best reported fully 4-bit results across reasoning, instruction-following, and structured prediction benchmarks, while also improving candidate-search efficiency and remaining compatible with multiple quantization datatypes.