Adaptive Data-Free Quantization

• URL: http://arxiv.org/abs/2303.06869v3
• Date: Mon, 20 Mar 2023 12:24:58 GMT
• Title: Adaptive Data-Free Quantization
• Authors: Biao Qian, Yang Wang, Richang Hong, Meng Wang
• Abstract summary: Data-free quantization (DFQ) recovers the performance of quantized network (Q) without the original data. DFQ generates the fake sample via a generator (G) by learning from full-precision network (P) We propose an Adaptive Data-Free Quantization (AdaDFQ) method, which revisits DFQ from a zero-sum game perspective.
• Score: 44.00726062583708
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Data-free quantization (DFQ) recovers the performance of quantized network (Q) without the original data, but generates the fake sample via a generator (G) by learning from full-precision network (P), which, however, is totally independent of Q, overlooking the adaptability of the knowledge from generated samples, i.e., informative or not to the learning process of Q, resulting into the overflow of generalization error. Building on this, several critical questions -- how to measure the sample adaptability to Q under varied bit-width scenarios? whether the largest adaptability is the best? how to generate the samples with adaptive adaptability to improve Q's generalization? To answer the above questions, in this paper, we propose an Adaptive Data-Free Quantization (AdaDFQ) method, which revisits DFQ from a zero-sum game perspective upon the sample adaptability between two players -- a generator and a quantized network. Following this viewpoint, we further define the disagreement and agreement samples to form two boundaries, where the margin is optimized to adaptively regulate the adaptability of generated samples to Q, so as to address the over-and-under fitting issues. Our AdaDFQ reveals: 1) the largest adaptability is NOT the best for sample generation to benefit Q's generalization; 2) the knowledge of the generated sample should not be informative to Q only, but also related to the category and distribution information of the training data for P. The theoretical and empirical analysis validate the advantages of AdaDFQ over the state-of-the-arts. Our code is available at https://github.com/hfutqian/AdaDFQ.

Related papers

• QASnowball: An Iterative Bootstrapping Framework for High-Quality Question-Answering Data Generation [67.27999343730224]
  We introduce an iterative bootstrapping framework for QA data augmentation (named QASnowball) QASnowball can iteratively generate large-scale high-quality QA data based on a seed set of supervised examples. We conduct experiments in the high-resource English scenario and the medium-resource Chinese scenario, and the experimental results show that the data generated by QASnowball can facilitate QA models.
  arXiv  Detail & Related papers  (2023-09-19T05:20:36Z)
• Rethinking Data-Free Quantization as a Zero-Sum Game [44.00726062583708]
  Data- quantization (DFQ) recovers the performance of quantized network (Q) without accessing the real data. DFQ generates a fake sample via a generator (G) by learning from full-precision network (P) instead. We propose an Adaptability-free Sample Generation (AdaSG) method to generate samples with desirable adaptability.
  arXiv  Detail & Related papers  (2023-02-19T13:22:40Z)
• Improving Question Answering with Generation of NQ-like Questions [12.276281998447079]
  Question Answering (QA) systems require a large amount of annotated data which is costly and time-consuming to gather. We propose an algorithm to automatically generate shorter questions resembling day-to-day human communication in the Natural Questions (NQ) dataset from longer trivia questions in Quizbowl (QB) dataset.
  arXiv  Detail & Related papers  (2022-10-12T21:36:20Z)
• Improving Unsupervised Question Answering via Summarization-Informed Question Generation [47.96911338198302]
  Question Generation (QG) is the task of generating a plausible question for a passage, answer> pair. We make use of freely available news summary data, transforming declarative sentences into appropriate questions using dependency parsing, named entity recognition and semantic role labeling. The resulting questions are then combined with the original news articles to train an end-to-end neural QG model.
  arXiv  Detail & Related papers  (2021-09-16T13:08:43Z)
• A New Score for Adaptive Tests in Bayesian and Credal Networks [64.80185026979883]
  A test is adaptive when its sequence and number of questions is dynamically tuned on the basis of the estimated skills of the taker. We present an alternative family of scores, based on the mode of the posterior probabilities, and hence easier to explain.
  arXiv  Detail & Related papers  (2021-05-25T20:35:42Z)
• Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs [62.71505254770827]
  We propose a conditional variational autoencoder (HCVAE) for generating QA pairs given unstructured texts as contexts. Our model obtains impressive performance gains over all baselines on both tasks, using only a fraction of data for training.
  arXiv  Detail & Related papers  (2020-05-28T08:26:06Z)
• Template-Based Question Generation from Retrieved Sentences for Improved Unsupervised Question Answering [98.48363619128108]
  We propose an unsupervised approach to training QA models with generated pseudo-training data. We show that generating questions for QA training by applying a simple template on a related, retrieved sentence rather than the original context sentence improves downstream QA performance.
  arXiv  Detail & Related papers  (2020-04-24T17:57:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.