Plant 'n' Seek: Can You Find the Winning Ticket?

• URL: http://arxiv.org/abs/2111.11153v1
• Date: Mon, 22 Nov 2021 12:32:25 GMT
• Title: Plant 'n' Seek: Can You Find the Winning Ticket?
• Authors: Jonas Fischer, Rebekka Burkholz
• Abstract summary: Lottery ticket hypothesis has sparked the rapid development of pruning algorithms that perform structure learning. We hand-craft extremely sparse network topologies, plant them in large neural networks, and evaluate state-of-the-art lottery ticket pruning methods.
• Score: 6.85316573653194
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that perform structure learning by identifying a sparse subnetwork of a large randomly initialized neural network. The existence of such 'winning tickets' has been proven theoretically but at suboptimal sparsity levels. Contemporary pruning algorithms have furthermore been struggling to identify sparse lottery tickets for complex learning tasks. Is this suboptimal sparsity merely an artifact of existence proofs and algorithms or a general limitation of the pruning approach? And, if very sparse tickets exist, are current algorithms able to find them or are further improvements needed to achieve effective network compression? To answer these questions systematically, we derive a framework to plant and hide target architectures within large randomly initialized neural networks. For three common challenges in machine learning, we hand-craft extremely sparse network topologies, plant them in large neural networks, and evaluate state-of-the-art lottery ticket pruning methods. We find that current limitations of pruning algorithms to identify extremely sparse tickets are likely of algorithmic rather than fundamental nature and anticipate that our planting framework will facilitate future developments of efficient pruning algorithms, as we have addressed the issue of missing baselines in the field raised by Frankle et al.

Related papers

• Finding Strong Lottery Ticket Networks with Genetic Algorithms [3.1267592104279776]
  According to the Strong Lottery Ticket Hypothesis, every sufficiently large neural network with randomly weights contains a sub-network which already performs as well for a given task as the trained super-network. We present the first approach based on a genetic algorithm to find such strong lottery ticket sub-networks without training or otherwise computing any gradient.
  arXiv  Detail & Related papers  (2024-11-07T12:35:35Z)
• The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
  We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples. In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
  arXiv  Detail & Related papers  (2023-03-17T02:01:11Z)
• Sparsity May Cry: Let Us Fail (Current) Sparse Neural Networks Together! [100.19080749267316]
  "Sparsity May Cry" Benchmark (SMC-Bench) is a collection of carefully-curated 4 diverse tasks with 10 datasets. SMC-Bench is designed to favor and encourage the development of more scalable and generalizable sparse algorithms.
  arXiv  Detail & Related papers  (2023-03-03T18:47:21Z)
• Rare Gems: Finding Lottery Tickets at Initialization [21.130411799740532]
  Large neural networks can be pruned to a small fraction of their original size. Current algorithms for finding trainable networks fail simple baseline comparisons. Finding lottery tickets that train to better accuracy compared to simple baselines remains an open problem.
  arXiv  Detail & Related papers  (2022-02-24T10:28:56Z)
• Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets [127.56361320894861]
  Lottery ticket hypothesis (LTH) has shown that dense models contain highly sparseworks (i.e., winning tickets) that can be trained in isolation to match full accuracy. In this paper, we demonstrate the first positive result that a structurally sparse winning ticket can be effectively found in general. Specifically, we first "re-fill" pruned elements back in some channels deemed to be important, and then "re-group" non-zero elements to create flexible group-wise structural patterns.
  arXiv  Detail & Related papers  (2022-02-09T21:33:51Z)
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks [79.74580058178594]
  We analyze the performance of training a pruned neural network by analyzing the geometric structure of the objective function. We show that the convex region near a desirable model with guaranteed generalization enlarges as the neural network model is pruned.
  arXiv  Detail & Related papers  (2021-10-12T01:11:07Z)
• Juvenile state hypothesis: What we can learn from lottery ticket hypothesis researches? [1.701869491238765]
  Original lottery ticket hypothesis performs pruning and weight resetting after training convergence. We propose a strategy that combines the idea of neural network structure search with a pruning algorithm to alleviate this problem.
  arXiv  Detail & Related papers  (2021-09-08T18:22:00Z)
• Towards Optimally Efficient Tree Search with Deep Learning [76.64632985696237]
  This paper investigates the classical integer least-squares problem which estimates signals integer from linear models. The problem is NP-hard and often arises in diverse applications such as signal processing, bioinformatics, communications and machine learning. We propose a general hyper-accelerated tree search (HATS) algorithm by employing a deep neural network to estimate the optimal estimation for the underlying simplified memory-bounded A* algorithm.
  arXiv  Detail & Related papers  (2021-01-07T08:00:02Z)
• Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot [55.37967301483917]
  Conventional wisdom of pruning algorithms suggests that pruning methods exploit information from training data to find goodworks. In this paper, we conduct sanity checks for the above beliefs on several recent unstructured pruning methods. We propose a series of simple emphdata-independent prune ratios for each layer, and randomly prune each layer accordingly to get a subnetwork.
  arXiv  Detail & Related papers  (2020-09-22T17:36:17Z)
• Pruning neural networks without any data by iteratively conserving synaptic flow [27.849332212178847]
  Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy. Recent works have identified, through an expensive sequence of training and pruning cycles, the existence of winning lottery tickets or sparse trainableworks. We provide an affirmative answer to this question through theory driven algorithm design.
  arXiv  Detail & Related papers  (2020-06-09T19:21:57Z)
• NeuroFabric: Identifying Ideal Topologies for Training A Priori Sparse Networks [2.398608007786179]
  Long training times of deep neural networks are a bottleneck in machine learning research. We provide a theoretical foundation for the choice of intra-layer topology. We show that seemingly similar topologies can often have a large difference in attainable accuracy.
  arXiv  Detail & Related papers  (2020-02-19T18:29:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.