How Does Calibration Data Affect the Post-training Pruning and
Quantization of Large Language Models?
- URL: http://arxiv.org/abs/2311.09755v1
- Date: Thu, 16 Nov 2023 10:30:00 GMT
- Title: How Does Calibration Data Affect the Post-training Pruning and
Quantization of Large Language Models?
- Authors: Miles Williams, Nikolaos Aletras
- Abstract summary: Pruning and quantization form the foundation of model compression for neural networks.
We present the first extensive empirical study on the effect of calibration data upon model compression methods.
- Score: 42.652021176354644
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Pruning and quantization form the foundation of model compression for neural
networks, enabling efficient inference for large language models (LLMs).
Recently, various quantization and pruning techniques have demonstrated
state-of-the-art performance in a post-training setting. They rely upon
calibration data, a small set of unlabeled examples, to generate layer
activations. However, no prior work has systematically investigated how the
calibration data impacts the effectiveness of model compression methods. In
this paper, we present the first extensive empirical study on the effect of
calibration data upon LLM performance. We trial a variety of pruning and
quantization methods, tasks, models, and datasets. Surprisingly, we find
substantial variations in downstream task performance, contrasting existing
work that suggests a greater level of robustness to the calibration data.
Finally, we make a series of recommendations for the effective use of
calibration data in LLM quantization and pruning.
Related papers
- MetaAug: Meta-Data Augmentation for Post-Training Quantization [32.02377559968568]
Post-Training Quantization (PTQ) has received significant attention because it requires only a small set of calibration data to quantize a full-precision model.
We propose a novel meta-learning based approach to enhance the performance of post-training quantization.
arXiv Detail & Related papers (2024-07-20T02:18:51Z) - On Task Performance and Model Calibration with Supervised and
Self-Ensembled In-Context Learning [71.44986275228747]
In-context learning (ICL) has become an efficient approach propelled by the recent advancements in large language models (LLMs)
However, both paradigms are prone to suffer from the critical problem of overconfidence (i.e., miscalibration)
arXiv Detail & Related papers (2023-12-21T11:55:10Z) - CATfOOD: Counterfactual Augmented Training for Improving Out-of-Domain
Performance and Calibration [59.48235003469116]
We show that data augmentation consistently enhances OOD performance.
We also show that CF augmented models which are easier to calibrate also exhibit much lower entropy when assigning importance.
arXiv Detail & Related papers (2023-09-14T16:16:40Z) - Towards Foundation Models for Scientific Machine Learning:
Characterizing Scaling and Transfer Behavior [32.74388989649232]
We study how pre-training could be used for scientific machine learning (SciML) applications.
We find that fine-tuning these models yields more performance gains as model size increases.
arXiv Detail & Related papers (2023-06-01T00:32:59Z) - To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs.
We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting.
Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
arXiv Detail & Related papers (2023-05-22T17:02:15Z) - Post-training Model Quantization Using GANs for Synthetic Data
Generation [57.40733249681334]
We investigate the use of synthetic data as a substitute for the calibration with real data for the quantization method.
We compare the performance of models quantized using data generated by StyleGAN2-ADA and our pre-trained DiStyleGAN, with quantization using real data and an alternative data generation method based on fractal images.
arXiv Detail & Related papers (2023-05-10T11:10:09Z) - On the Importance of Calibration in Semi-supervised Learning [13.859032326378188]
State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data.
We introduce a family of new SSL models that optimize for calibration and demonstrate their effectiveness across standard vision benchmarks.
arXiv Detail & Related papers (2022-10-10T15:41:44Z) - How Training Data Impacts Performance in Learning-based Control [67.7875109298865]
This paper derives an analytical relationship between the density of the training data and the control performance.
We formulate a quality measure for the data set, which we refer to as $rho$-gap.
We show how the $rho$-gap can be applied to a feedback linearizing control law.
arXiv Detail & Related papers (2020-05-25T12:13:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.