Self-Supervised GAN Compression

• URL: http://arxiv.org/abs/2007.01491v2
• Date: Sun, 12 Jul 2020 16:43:57 GMT
• Title: Self-Supervised GAN Compression
• Authors: Chong Yu, Jeff Pool
• Abstract summary: We show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.
• Score: 32.21713098893454
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep learning's success has led to larger and larger models to handle more and more complex tasks; trained models can contain millions of parameters. These large models are compute- and memory-intensive, which makes it a challenge to deploy them with minimized latency, throughput, and storage requirements. Some model compression methods have been successfully applied to image classification and detection or language models, but there has been very little work compressing generative adversarial networks (GANs) performing complex tasks. In this paper, we show that a standard model compression technique, weight pruning, cannot be applied to GANs using existing methods. We then develop a self-supervised compression technique which uses the trained discriminator to supervise the training of a compressed generator. We show that this framework has a compelling performance to high degrees of sparsity, can be easily applied to new tasks and models, and enables meaningful comparisons between different pruning granularities.

Related papers

• MCNC: Manifold Constrained Network Compression [21.70510507535041]
  We present MCNC as a novel model compression method that constrains the parameter space to low-dimensional pre-defined and frozen nonlinear manifold. We show that our method, MCNC, significantly outperforms state-of-the-art baselines in terms of compression, accuracy, and/or model reconstruction time.
  arXiv  Detail & Related papers  (2024-06-27T16:17:26Z)
• Tiny Models are the Computational Saver for Large Models [1.8350044465969415]
  This paper introduces TinySaver, an early-exit-like dynamic model compression approach which employs tiny models to substitute large models adaptively. Our evaluation of this approach in ImageNet-1k classification demonstrates its potential to reduce the number of compute operations by up to 90%, with only negligible losses in performance.
  arXiv  Detail & Related papers  (2024-03-26T14:14:30Z)
• Activations and Gradients Compression for Model-Parallel Training [85.99744701008802]
  We study how simultaneous compression of activations and gradients in model-parallel distributed training setup affects convergence. We find that gradients require milder compression rates than activations. Experiments also show that models trained with TopK perform well only when compression is also applied during inference.
  arXiv  Detail & Related papers  (2024-01-15T15:54:54Z)
• Retrieval-based Knowledge Transfer: An Effective Approach for Extreme Large Language Model Compression [64.07696663255155]
  Large-scale pre-trained language models (LLMs) have demonstrated exceptional performance in various natural language processing (NLP) tasks. However, the massive size of these models poses huge challenges for their deployment in real-world applications. We introduce a novel compression paradigm called Retrieval-based Knowledge Transfer (RetriKT) which effectively transfers the knowledge of LLMs to extremely small-scale models.
  arXiv  Detail & Related papers  (2023-10-24T07:58:20Z)
• Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition [7.450574974954803]
  Small memory bandwidth in smart devices prompts development of smaller Automatic Speech Recognition (ASR) models. Knowledge distillation (KD) is a popular model compression approach that has shown to achieve smaller model size. We propose a multi-stage progressive approach to compress the conformer transducer model using KD.
  arXiv  Detail & Related papers  (2022-10-01T02:23:00Z)
• What do Compressed Large Language Models Forget? Robustness Challenges in Model Compression [68.82486784654817]
  We study two popular model compression techniques including knowledge distillation and pruning. We show that compressed models are significantly less robust than their PLM counterparts on adversarial test sets. We develop a regularization strategy for model compression based on sample uncertainty.
  arXiv  Detail & Related papers  (2021-10-16T00:20:04Z)
• NAS-BERT: Task-Agnostic and Adaptive-Size BERT Compression with Neural Architecture Search [100.71365025972258]
  We propose NAS-BERT, an efficient method for BERT compression. NAS-BERT trains a big supernet on a search space and outputs multiple compressed models with adaptive sizes and latency. Experiments on GLUE and SQuAD benchmark datasets demonstrate that NAS-BERT can find lightweight models with better accuracy than previous approaches.
  arXiv  Detail & Related papers  (2021-05-30T07:20:27Z)
• GAN Slimming: All-in-One GAN Compression by A Unified Optimization Framework [94.26938614206689]
  We propose the first unified optimization framework combining multiple compression means for GAN compression, dubbed GAN Slimming. We apply GS to compress CartoonGAN, a state-of-the-art style transfer network, by up to 47 times, with minimal visual quality degradation.
  arXiv  Detail & Related papers  (2020-08-25T14:39:42Z)
• Dynamic Model Pruning with Feedback [64.019079257231]
  We propose a novel model compression method that generates a sparse trained model without additional overhead. We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models.
  arXiv  Detail & Related papers  (2020-06-12T15:07:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.