Winning Lottery Tickets in Deep Generative Models

• URL: http://arxiv.org/abs/2010.02350v2
• Date: Fri, 29 Jan 2021 18:44:21 GMT
• Title: Winning Lottery Tickets in Deep Generative Models
• Authors: Neha Mukund Kalibhat, Yogesh Balaji, Soheil Feizi
• Abstract summary: We show the existence of winning tickets in deep generative models such as GANs and VAEs. We also demonstrate the transferability of winning tickets across different generative models.
• Score: 64.79920299421255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The lottery ticket hypothesis suggests that sparse, sub-networks of a given neural network, if initialized properly, can be trained to reach comparable or even better performance to that of the original network. Prior works in lottery tickets have primarily focused on the supervised learning setup, with several papers proposing effective ways of finding "winning tickets" in classification problems. In this paper, we confirm the existence of winning tickets in deep generative models such as GANs and VAEs. We show that the popular iterative magnitude pruning approach (with late rewinding) can be used with generative losses to find the winning tickets. This approach effectively yields tickets with sparsity up to 99% for AutoEncoders, 93% for VAEs and 89% for GANs on CIFAR and Celeb-A datasets. We also demonstrate the transferability of winning tickets across different generative models (GANs and VAEs) sharing the same architecture, suggesting that winning tickets have inductive biases that could help train a wide range of deep generative models. Furthermore, we show the practical benefits of lottery tickets in generative models by detecting tickets at very early stages in training called "early-bird tickets". Through early-bird tickets, we can achieve up to 88% reduction in floating-point operations (FLOPs) and 54% reduction in training time, making it possible to train large-scale generative models over tight resource constraints. These results out-perform existing early pruning methods like SNIP (Lee, Ajanthan, and Torr 2019) and GraSP (Wang, Zhang, and Grosse 2020). Our findings shed light towards existence of proper network initializations that could improve convergence and stability of generative models.

Related papers

• COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks [5.956029437413275]
  This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch.
  arXiv  Detail & Related papers  (2022-12-24T16:38:59Z)
• Can We Find Strong Lottery Tickets in Generative Models? [24.405555822170896]
  We find strong lottery tickets in generative models that achieve good generative performance without any weight update. To the best of our knowledge, we are the first to show the existence of strong lottery tickets in generative models and provide an algorithm to find it.
  arXiv  Detail & Related papers  (2022-12-16T07:20:28Z)
• Dual Lottery Ticket Hypothesis [71.95937879869334]
  Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark. We propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH.
  arXiv  Detail & Related papers  (2022-03-08T18:06:26Z)
• Efficient Lottery Ticket Finding: Less Data is More [87.13642800792077]
  Lottery ticket hypothesis (LTH) reveals existence of winning tickets (sparse but criticalworks) for dense networks. Finding winning tickets requires burdensome computations in the train-prune-retrain process. This paper explores a new perspective on finding lottery tickets more efficiently, by doing so only with a specially selected subset of data.
  arXiv  Detail & Related papers  (2021-06-06T19:58:17Z)
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization [65.23099004725461]
  We study such a collection of tickets, which is referred to as "winning tickets", in extremely over-parametrized models. We observe that at certain compression ratios, generalization performance of the winning tickets can not only match, but also exceed that of the full model.
  arXiv  Detail & Related papers  (2021-05-25T15:10:05Z)
• The Elastic Lottery Ticket Hypothesis [106.79387235014379]
  Lottery Ticket Hypothesis raises keen attention to identifying sparse trainableworks or winning tickets. The most effective method to identify such winning tickets is still Iterative Magnitude-based Pruning. We propose a variety of strategies to tweak the winning tickets found from different networks of the same model family.
  arXiv  Detail & Related papers  (2021-03-30T17:53:45Z)
• Good Students Play Big Lottery Better [84.6111281091602]
  Lottery ticket hypothesis suggests that a dense neural network contains a sparse sub-network that can match the test accuracy of the original dense net. Recent studies demonstrate that a sparse sub-network can still be obtained by using a rewinding technique. This paper proposes a new, simpler and yet powerful technique for re-training the sub-network, called "Knowledge Distillation ticket" (KD ticket)
  arXiv  Detail & Related papers  (2021-01-08T23:33:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.