Diversifying Sample Generation for Accurate Data-Free Quantization
- URL: http://arxiv.org/abs/2103.01049v1
- Date: Mon, 1 Mar 2021 14:46:02 GMT
- Title: Diversifying Sample Generation for Accurate Data-Free Quantization
- Authors: Xiangguo Zhang, Haotong Qin, Yifu Ding, Ruihao Gong, Qinghua Yan,
Renshuai Tao, Yuhang Li, Fengwei Yu, Xianglong Liu
- Abstract summary: We propose Diverse Sample Generation (DSG) scheme to mitigate the adverse effects caused by homogenization.
Our scheme is versatile and even able to be applied to the state-of-the-art post-training quantization method like AdaRound.
- Score: 35.38029335993735
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Quantization has emerged as one of the most prevalent approaches to compress
and accelerate neural networks. Recently, data-free quantization has been
widely studied as a practical and promising solution. It synthesizes data for
calibrating the quantized model according to the batch normalization (BN)
statistics of FP32 ones and significantly relieves the heavy dependency on real
training data in traditional quantization methods. Unfortunately, we find that
in practice, the synthetic data identically constrained by BN statistics
suffers serious homogenization at both distribution level and sample level and
further causes a significant performance drop of the quantized model. We
propose Diverse Sample Generation (DSG) scheme to mitigate the adverse effects
caused by homogenization. Specifically, we slack the alignment of feature
statistics in the BN layer to relax the constraint at the distribution level
and design a layerwise enhancement to reinforce specific layers for different
data samples. Our DSG scheme is versatile and even able to be applied to the
state-of-the-art post-training quantization method like AdaRound. We evaluate
the DSG scheme on the large-scale image classification task and consistently
obtain significant improvements over various network architectures and
quantization methods, especially when quantized to lower bits (e.g., up to 22%
improvement on W4A4). Moreover, benefiting from the enhanced diversity, models
calibrated by synthetic data perform close to those calibrated by real data and
even outperform them on W4A4.
Related papers
- Data Generation for Hardware-Friendly Post-Training Quantization [3.3998740964877463]
Zero-shot quantization (ZSQ) using synthetic data is a key approach for post-training quantization (PTQ) under privacy and security constraints.
Existing data generation methods often struggle to effectively generate data suitable for hardware-friendly quantization.
We propose Data Generation for Hardware-friendly quantization (DGH), a novel method that addresses these gaps.
arXiv Detail & Related papers (2024-10-29T15:08:50Z) - EDA-DM: Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models [4.21216544443537]
Quantization can effectively reduce model complexity, and post-training quantization (PTQ) is highly promising for compressing and accelerating diffusion models.
Existing PTQ methods for diffusion models suffer from distribution mismatch issues at both calibration sample level and reconstruction output level.
We propose Enhanced Distribution Alignment for Post-Training Quantization of Diffusion Models (EDA-DM) to address the above issues.
arXiv Detail & Related papers (2024-01-09T14:42:49Z) - Post-training Model Quantization Using GANs for Synthetic Data
Generation [57.40733249681334]
We investigate the use of synthetic data as a substitute for the calibration with real data for the quantization method.
We compare the performance of models quantized using data generated by StyleGAN2-ADA and our pre-trained DiStyleGAN, with quantization using real data and an alternative data generation method based on fractal images.
arXiv Detail & Related papers (2023-05-10T11:10:09Z) - Q-Diffusion: Quantizing Diffusion Models [52.978047249670276]
Post-training quantization (PTQ) is considered a go-to compression method for other tasks.
We propose a novel PTQ method specifically tailored towards the unique multi-timestep pipeline and model architecture.
We show that our proposed method is able to quantize full-precision unconditional diffusion models into 4-bit while maintaining comparable performance.
arXiv Detail & Related papers (2023-02-08T19:38:59Z) - ScoreMix: A Scalable Augmentation Strategy for Training GANs with
Limited Data [93.06336507035486]
Generative Adversarial Networks (GANs) typically suffer from overfitting when limited training data is available.
We present ScoreMix, a novel and scalable data augmentation approach for various image synthesis tasks.
arXiv Detail & Related papers (2022-10-27T02:55:15Z) - ClusterQ: Semantic Feature Distribution Alignment for Data-Free
Quantization [111.12063632743013]
We propose a new and effective data-free quantization method termed ClusterQ.
To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics.
We also incorporate the intra-class variance to solve class-wise mode collapse.
arXiv Detail & Related papers (2022-04-30T06:58:56Z) - Diverse Sample Generation: Pushing the Limit of Data-free Quantization [85.95032037447454]
This paper presents a generic Diverse Sample Generation scheme for the generative data-free post-training quantization and quantization-aware training.
For large-scale image classification tasks, our DSG can consistently outperform existing data-free quantization methods.
arXiv Detail & Related papers (2021-09-01T07:06:44Z) - Zero-shot Adversarial Quantization [11.722728148523366]
We propose a zero-shot adversarial quantization (ZAQ) framework, facilitating effective discrepancy estimation and knowledge transfer.
This is achieved by a novel two-level discrepancy modeling to drive a generator to synthesize informative and diverse data examples.
We conduct extensive experiments on three fundamental vision tasks, demonstrating the superiority of ZAQ over the strong zero-shot baselines.
arXiv Detail & Related papers (2021-03-29T01:33:34Z) - Optimal Gradient Quantization Condition for Communication-Efficient
Distributed Training [99.42912552638168]
Communication of gradients is costly for training deep neural networks with multiple devices in computer vision applications.
In this work, we deduce the optimal condition of both the binary and multi-level gradient quantization for textbfANY gradient distribution.
Based on the optimal condition, we develop two novel quantization schemes: biased BinGrad and unbiased ORQ for binary and multi-level gradient quantization respectively.
arXiv Detail & Related papers (2020-02-25T18:28:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.