MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing

• URL: http://arxiv.org/abs/2011.09899v3
• Date: Wed, 26 Jan 2022 22:25:41 GMT
• Title: MixMix: All You Need for Data-Free Compression Are Feature and Data Mixing
• Authors: Yuhang Li, Feng Zhu, Ruihao Gong, Mingzhu Shen, Xin Dong, Fengwei Yu, Shaoqing Lu, Shi Gu
• Abstract summary: We propose MixMix to overcome the difficulties of generalizability and inexact inversion. We prove the effectiveness of MixMix from both theoretical and empirical perspectives. MixMix achieves up to 4% and 20% accuracy uplift on quantization and pruning, respectively, compared to existing data-free compression work.
• Score: 30.14401315979937
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: User data confidentiality protection is becoming a rising challenge in the present deep learning research. Without access to data, conventional data-driven model compression faces a higher risk of performance degradation. Recently, some works propose to generate images from a specific pretrained model to serve as training data. However, the inversion process only utilizes biased feature statistics stored in one model and is from low-dimension to high-dimension. As a consequence, it inevitably encounters the difficulties of generalizability and inexact inversion, which leads to unsatisfactory performance. To address these problems, we propose MixMix based on two simple yet effective techniques: (1) Feature Mixing: utilizes various models to construct a universal feature space for generalized inversion; (2) Data Mixing: mixes the synthesized images and labels to generate exact label information. We prove the effectiveness of MixMix from both theoretical and empirical perspectives. Extensive experiments show that MixMix outperforms existing methods on the mainstream compression tasks, including quantization, knowledge distillation, and pruning. Specifically, MixMix achieves up to 4% and 20% accuracy uplift on quantization and pruning, respectively, compared to existing data-free compression work.

Related papers

• SUMix: Mixup with Semantic and Uncertain Information [41.99721365685618]
  Mixup data augmentation approaches have been applied for various tasks of deep learning. We propose a novel approach named SUMix to learn the mixing ratio as well as the uncertainty for the mixed samples during the training process.
  arXiv  Detail & Related papers  (2024-07-10T16:25:26Z)
• RC-Mixup: A Data Augmentation Strategy against Noisy Data for Regression Tasks [27.247270530020664]
  We study the problem of robust data augmentation for regression tasks in the presence of noisy data. C-Mixup is more selective in which samples to mix based on their label distances for better regression performance. We propose RC-Mixup, which tightly integrates C-Mixup with multi-round robust training methods for a synergistic effect.
  arXiv  Detail & Related papers  (2024-05-28T08:02:42Z)
• Data Mixing Laws: Optimizing Data Mixtures by Predicting Language Modeling Performance [55.872926690722714]
  We study the predictability of model performance regarding the mixture proportions in function forms. We propose nested use of the scaling laws of training steps, model sizes, and our data mixing law. Our method effectively optimize the training mixture of a 1B model trained for 100B tokens in RedPajama.
  arXiv  Detail & Related papers  (2024-03-25T17:14:00Z)
• TransformMix: Learning Transformation and Mixing Strategies from Data [20.79680733590554]
  We propose an automated approach, TransformMix, to learn better transformation and mixing augmentation strategies from data. We demonstrate the effectiveness of TransformMix on multiple datasets in transfer learning, classification, object detection, and knowledge distillation settings.
  arXiv  Detail & Related papers  (2024-03-19T04:36:41Z)
• The Benefits of Mixup for Feature Learning [117.93273337740442]
  We first show that Mixup using different linear parameters for features and labels can still achieve similar performance to standard Mixup. We consider a feature-noise data model and show that Mixup training can effectively learn the rare features from its mixture with the common features. In contrast, standard training can only learn the common features but fails to learn the rare features, thus suffering from bad performance.
  arXiv  Detail & Related papers  (2023-03-15T08:11:47Z)
• DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification [56.817386699291305]
  This paper proposes a simple yet effective data augmentation approach termed DoubleMix. DoubleMix first generates several perturbed samples for each training data. It then uses the perturbed data and original data to carry out a two-step in the hidden space of neural models.
  arXiv  Detail & Related papers  (2022-09-12T15:01:04Z)
• RandoMix: A mixed sample data augmentation method with multiple mixed modes [12.466162659083697]
  RandoMix is a mixed-sample data augmentation method designed to address robustness and diversity challenges. We evaluate the effectiveness of RandoMix on diverse datasets, including CIFAR-10/100, Tiny-ImageNet, ImageNet, and Google Speech Commands.
  arXiv  Detail & Related papers  (2022-05-18T05:31:36Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• ResizeMix: Mixing Data with Preserved Object Information and True Labels [57.00554495298033]
  We study the importance of the saliency information for mixing data, and find that the saliency information is not so necessary for promoting the augmentation performance. We propose a more effective but very easily implemented method, namely ResizeMix.
  arXiv  Detail & Related papers  (2020-12-21T03:43:13Z)
• SnapMix: Semantically Proportional Mixing for Augmenting Fine-grained Data [124.95585891086894]
  Proposal is called Semantically Proportional Mixing (SnapMix) It exploits class activation map (CAM) to lessen the label noise in augmenting fine-grained data. Our method consistently outperforms existing mixed-based approaches.
  arXiv  Detail & Related papers  (2020-12-09T03:37:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.