COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks

• URL: http://arxiv.org/abs/2212.12770v1
• Date: Sat, 24 Dec 2022 16:38:59 GMT
• Title: COLT: Cyclic Overlapping Lottery Tickets for Faster Pruning of Convolutional Neural Networks
• Authors: Md. Ismail Hossain, Mohammed Rakib, M. M. Lutfe Elahi, Nabeel Mohammed and Shafin Rahman
• Abstract summary: This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch.
• Score: 5.956029437413275
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pruning refers to the elimination of trivial weights from neural networks. The sub-networks within an overparameterized model produced after pruning are often called Lottery tickets. This research aims to generate winning lottery tickets from a set of lottery tickets that can achieve similar accuracy to the original unpruned network. We introduce a novel winning ticket called Cyclic Overlapping Lottery Ticket (COLT) by data splitting and cyclic retraining of the pruned network from scratch. We apply a cyclic pruning algorithm that keeps only the overlapping weights of different pruned models trained on different data segments. Our results demonstrate that COLT can achieve similar accuracies (obtained by the unpruned model) while maintaining high sparsities. We show that the accuracy of COLT is on par with the winning tickets of Lottery Ticket Hypothesis (LTH) and, at times, is better. Moreover, COLTs can be generated using fewer iterations than tickets generated by the popular Iterative Magnitude Pruning (IMP) method. In addition, we also notice COLTs generated on large datasets can be transferred to small ones without compromising performance, demonstrating its generalizing capability. We conduct all our experiments on Cifar-10, Cifar-100 & TinyImageNet datasets and report superior performance than the state-of-the-art methods.

Related papers

• Dual Lottery Ticket Hypothesis [71.95937879869334]
  Lottery Ticket Hypothesis (LTH) provides a novel view to investigate sparse network training and maintain its capacity. In this work, we regard the winning ticket from LTH as the subnetwork which is in trainable condition and its performance as our benchmark. We propose a simple sparse network training strategy, Random Sparse Network Transformation (RST), to substantiate our DLTH.
  arXiv  Detail & Related papers  (2022-03-08T18:06:26Z)
• Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets [127.56361320894861]
  Lottery ticket hypothesis (LTH) has shown that dense models contain highly sparseworks (i.e., winning tickets) that can be trained in isolation to match full accuracy. In this paper, we demonstrate the first positive result that a structurally sparse winning ticket can be effectively found in general. Specifically, we first "re-fill" pruned elements back in some channels deemed to be important, and then "re-group" non-zero elements to create flexible group-wise structural patterns.
  arXiv  Detail & Related papers  (2022-02-09T21:33:51Z)
• Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalization [65.23099004725461]
  We study such a collection of tickets, which is referred to as "winning tickets", in extremely over-parametrized models. We observe that at certain compression ratios, generalization performance of the winning tickets can not only match, but also exceed that of the full model.
  arXiv  Detail & Related papers  (2021-05-25T15:10:05Z)
• The Elastic Lottery Ticket Hypothesis [106.79387235014379]
  Lottery Ticket Hypothesis raises keen attention to identifying sparse trainableworks or winning tickets. The most effective method to identify such winning tickets is still Iterative Magnitude-based Pruning. We propose a variety of strategies to tweak the winning tickets found from different networks of the same model family.
  arXiv  Detail & Related papers  (2021-03-30T17:53:45Z)
• Good Students Play Big Lottery Better [84.6111281091602]
  Lottery ticket hypothesis suggests that a dense neural network contains a sparse sub-network that can match the test accuracy of the original dense net. Recent studies demonstrate that a sparse sub-network can still be obtained by using a rewinding technique. This paper proposes a new, simpler and yet powerful technique for re-training the sub-network, called "Knowledge Distillation ticket" (KD ticket)
  arXiv  Detail & Related papers  (2021-01-08T23:33:53Z)
• Winning Lottery Tickets in Deep Generative Models [64.79920299421255]
  We show the existence of winning tickets in deep generative models such as GANs and VAEs. We also demonstrate the transferability of winning tickets across different generative models.
  arXiv  Detail & Related papers  (2020-10-05T21:45:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.