Diverse Sample Generation: Pushing the Limit of Data-free Quantization

• URL: http://arxiv.org/abs/2109.00212v2
• Date: Fri, 3 Sep 2021 08:02:21 GMT
• Title: Diverse Sample Generation: Pushing the Limit of Data-free Quantization
• Authors: Haotong Qin, Yifu Ding, Xiangguo Zhang, Aoyu Li, Jiakai Wang, Xianglong Liu, Jiwen Lu
• Abstract summary: This paper presents a generic Diverse Sample Generation scheme for the generative data-free post-training quantization and quantization-aware training. For large-scale image classification tasks, our DSG can consistently outperform existing data-free quantization methods.
• Score: 85.95032037447454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, generative data-free quantization emerges as a practical approach that compresses the neural network to low bit-width without access to real data. It generates data to quantize the network by utilizing the batch normalization (BN) statistics of its full-precision counterpart. However, our study shows that in practice, the synthetic data completely constrained by BN statistics suffers severe homogenization at distribution and sample level, which causes serious accuracy degradation of the quantized network. This paper presents a generic Diverse Sample Generation (DSG) scheme for the generative data-free post-training quantization and quantization-aware training, to mitigate the detrimental homogenization. In our DSG, we first slack the statistics alignment for features in the BN layer to relax the distribution constraint. Then we strengthen the loss impact of the specific BN layer for different samples and inhibit the correlation among samples in the generation process, to diversify samples from the statistical and spatial perspective, respectively. Extensive experiments show that for large-scale image classification tasks, our DSG can consistently outperform existing data-free quantization methods on various neural architectures, especially under ultra-low bit-width (e.g., 22% gain under W4A4 setting). Moreover, data diversifying caused by our DSG brings a general gain in various quantization methods, demonstrating diversity is an important property of high-quality synthetic data for data-free quantization.

Related papers

• Data Generation for Hardware-Friendly Post-Training Quantization [3.3998740964877463]
  Zero-shot quantization (ZSQ) using synthetic data is a key approach for post-training quantization (PTQ) under privacy and security constraints. Existing data generation methods often struggle to effectively generate data suitable for hardware-friendly quantization. We propose Data Generation for Hardware-friendly quantization (DGH), a novel method that addresses these gaps.
  arXiv  Detail & Related papers  (2024-10-29T15:08:50Z)
• Post-training Model Quantization Using GANs for Synthetic Data Generation [57.40733249681334]
  We investigate the use of synthetic data as a substitute for the calibration with real data for the quantization method. We compare the performance of models quantized using data generated by StyleGAN2-ADA and our pre-trained DiStyleGAN, with quantization using real data and an alternative data generation method based on fractal images.
  arXiv  Detail & Related papers  (2023-05-10T11:10:09Z)
• LD-GAN: Low-Dimensional Generative Adversarial Network for Spectral Image Generation with Variance Regularization [72.4394510913927]
  Deep learning methods are state-of-the-art for spectral image (SI) computational tasks. GANs enable diverse augmentation by learning and sampling from the data distribution. GAN-based SI generation is challenging since the high-dimensionality nature of this kind of data hinders the convergence of the GAN training yielding to suboptimal generation. We propose a statistical regularization to control the low-dimensional representation variance for the autoencoder training and to achieve high diversity of samples generated with the GAN.
  arXiv  Detail & Related papers  (2023-04-29T00:25:02Z)
• Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks [89.28881869440433]
  This paper provides the first theoretical characterization of joint edge-model sparse learning for graph neural networks (GNNs) It proves analytically that both sampling important nodes and pruning neurons with the lowest-magnitude can reduce the sample complexity and improve convergence without compromising the test accuracy.
  arXiv  Detail & Related papers  (2023-02-06T16:54:20Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss [61.26793005355441]
  Cluster-Promoting Quantization (CPQ) finds the optimal quantization grids for neural networks. DropBits is a new bit-drop technique that revises the standard dropout regularization to randomly drop bits instead of neurons. We experimentally validate our method on various benchmark datasets and network architectures.
  arXiv  Detail & Related papers  (2021-09-05T15:15:07Z)
• Zero-shot Adversarial Quantization [11.722728148523366]
  We propose a zero-shot adversarial quantization (ZAQ) framework, facilitating effective discrepancy estimation and knowledge transfer. This is achieved by a novel two-level discrepancy modeling to drive a generator to synthesize informative and diverse data examples. We conduct extensive experiments on three fundamental vision tasks, demonstrating the superiority of ZAQ over the strong zero-shot baselines.
  arXiv  Detail & Related papers  (2021-03-29T01:33:34Z)
• Data-free mixed-precision quantization using novel sensitivity metric [6.031526641614695]
  We propose a novel sensitivity metric that considers the effect of quantization error on task loss and interaction with other layers. Our experiments show that the proposed metric better represents quantization sensitivity, and generated data are more feasible to be applied to mixed-precision quantization.
  arXiv  Detail & Related papers  (2021-03-18T07:23:21Z)
• Diversifying Sample Generation for Accurate Data-Free Quantization [35.38029335993735]
  We propose Diverse Sample Generation (DSG) scheme to mitigate the adverse effects caused by homogenization. Our scheme is versatile and even able to be applied to the state-of-the-art post-training quantization method like AdaRound.
  arXiv  Detail & Related papers  (2021-03-01T14:46:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.