Related papers: PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression

PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression

URL: http://arxiv.org/abs/2405.14852v2
Date: Thu, 30 May 2024 15:01:49 GMT
Title: PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM Compression
Authors: Vladimir Malinovskii, Denis Mazur, Ivan Ilin, Denis Kuznedelev, Konstantin Burlachenko, Kai Yi, Dan Alistarh, Peter Richtarik,
Abstract summary: State-of-the-art quantization methods include fine-tuning (part of) the compressed parameters over a limited amount of calibration data. We propose PV-Tuning - a representation-agnostic framework that generalizes and improves upon existing fine-tuning strategies.
Score: 31.30170080420504
License: http://creativecommons.org/licenses/by/4.0/
Abstract: There has been significant interest in "extreme" compression of large language models (LLMs), i.e., to 1-2 bits per parameter, which allows such models to be executed efficiently on resource-constrained devices. Existing work focused on improved one-shot quantization techniques and weight representations; yet, purely post-training approaches are reaching diminishing returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art quantization methods such as QuIP# and AQLM include fine-tuning (part of) the compressed parameters over a limited amount of calibration data; however, such fine-tuning techniques over compressed weights often make exclusive use of straight-through estimators (STE), whose performance is not well-understood in this setting. In this work, we question the use of STE for extreme LLM compression, showing that it can be sub-optimal, and perform a systematic study of quantization-aware fine-tuning strategies for LLMs. We propose PV-Tuning - a representation-agnostic framework that generalizes and improves upon existing fine-tuning strategies, and provides convergence guarantees in restricted cases. On the practical side, when used for 1-2 bit vector quantization, PV-Tuning outperforms prior techniques for highly-performant models such as Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal quantization for Llama 2 family models at 2 bits per parameter.