Rethinking Data-Free Quantization as a Zero-Sum Game

• URL: http://arxiv.org/abs/2302.09572v1
• Date: Sun, 19 Feb 2023 13:22:40 GMT
• Title: Rethinking Data-Free Quantization as a Zero-Sum Game
• Authors: Biao Qian, Yang Wang, Richang Hong and Meng Wang
• Abstract summary: Data- quantization (DFQ) recovers the performance of quantized network (Q) without accessing the real data. DFQ generates a fake sample via a generator (G) by learning from full-precision network (P) instead. We propose an Adaptability-free Sample Generation (AdaSG) method to generate samples with desirable adaptability.
• Score: 44.00726062583708
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-free quantization (DFQ) recovers the performance of quantized network (Q) without accessing the real data, but generates the fake sample via a generator (G) by learning from full-precision network (P) instead. However, such sample generation process is totally independent of Q, specialized as failing to consider the adaptability of the generated samples, i.e., beneficial or adversarial, over the learning process of Q, resulting into non-ignorable performance loss. Building on this, several crucial questions -- how to measure and exploit the sample adaptability to Q under varied bit-width scenarios? how to generate the samples with desirable adaptability to benefit the quantized network? -- impel us to revisit DFQ. In this paper, we answer the above questions from a game-theory perspective to specialize DFQ as a zero-sum game between two players -- a generator and a quantized network, and further propose an Adaptability-aware Sample Generation (AdaSG) method. Technically, AdaSG reformulates DFQ as a dynamic maximization-vs-minimization game process anchored on the sample adaptability. The maximization process aims to generate the sample with desirable adaptability, such sample adaptability is further reduced by the minimization process after calibrating Q for performance recovery. The Balance Gap is defined to guide the stationarity of the game process to maximally benefit Q. The theoretical analysis and empirical studies verify the superiority of AdaSG over the state-of-the-arts. Our code is available at https://github.com/hfutqian/AdaSG.

Related papers

• Test-Time Model Adaptation with Only Forward Passes [68.11784295706995]
  Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts. We propose a test-time Forward-Optimization Adaptation (FOA) method. FOA runs on quantized 8-bit ViT, outperforms gradient-based TENT on full-precision 32-bit ViT, and achieves an up to 24-fold memory reduction on ImageNet-C.
  arXiv  Detail & Related papers  (2024-04-02T05:34:33Z)
• Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splitting [13.270381125055275]
  We propose a coarse & fine weight splitting (CFWS) method to reduce quantization error of weight. We develop an improved KL metric to determine optimal quantization scales for activation. For example, the quantized RepVGG-A1 model exhibits a mere 0.3% accuracy loss.
  arXiv  Detail & Related papers  (2023-12-17T02:31:20Z)
• EQ-Net: Elastic Quantization Neural Networks [15.289359357583079]
  Elastic Quantization Neural Networks (EQ-Net) aims to train a robust weight-sharing quantization supernet. We propose an elastic quantization space (including elastic bit-width, granularity, and symmetry) to adapt to various mainstream quantitative forms. We incorporate genetic algorithms and the proposed Conditional Quantization-Aware Conditional Accuracy Predictor (CQAP) as an estimator to quickly search mixed-precision quantized neural networks in supernet.
  arXiv  Detail & Related papers  (2023-08-15T08:57:03Z)
• Adaptive Data-Free Quantization [44.00726062583708]
  Data-free quantization (DFQ) recovers the performance of quantized network (Q) without the original data. DFQ generates the fake sample via a generator (G) by learning from full-precision network (P) We propose an Adaptive Data-Free Quantization (AdaDFQ) method, which revisits DFQ from a zero-sum game perspective.
  arXiv  Detail & Related papers  (2023-03-13T05:37:40Z)
• ClusterQ: Semantic Feature Distribution Alignment for Data-Free Quantization [111.12063632743013]
  We propose a new and effective data-free quantization method termed ClusterQ. To obtain high inter-class separability of semantic features, we cluster and align the feature distribution statistics. We also incorporate the intra-class variance to solve class-wise mode collapse.
  arXiv  Detail & Related papers  (2022-04-30T06:58:56Z)
• Deep Non-Crossing Quantiles through the Partial Derivative [0.6299766708197883]
  Quantile Regression provides a way to approximate a single conditional quantile. Minimisation of the QR-loss function does not guarantee non-crossing quantiles. We propose a generic deep learning algorithm for predicting an arbitrary number of quantiles.
  arXiv  Detail & Related papers  (2022-01-30T15:35:21Z)
• Learning a Unified Sample Weighting Network for Object Detection [113.98404690619982]
  Region sampling or weighting is significantly important to the success of modern region-based object detectors. We argue that sample weighting should be data-dependent and task-dependent. We propose a unified sample weighting network to predict a sample's task weights.
  arXiv  Detail & Related papers  (2020-06-11T16:19:16Z)
• Feature Quantization Improves GAN Training [126.02828112121874]
  Feature Quantization (FQ) for the discriminator embeds both true and fake data samples into a shared discrete space. Our method can be easily plugged into existing GAN models, with little computational overhead in training.
  arXiv  Detail & Related papers  (2020-04-05T04:06:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.