Extracting Effective Subnetworks with Gumebel-Softmax

• URL: http://arxiv.org/abs/2202.12986v1
• Date: Fri, 25 Feb 2022 21:31:30 GMT
• Title: Extracting Effective Subnetworks with Gumebel-Softmax
• Authors: Robin Dupont, Mohammed Amine Alaoui, Hichem Sahbi, Alice Lebois
• Abstract summary: We devise an alternative pruning method that allows extracting effective pruningworks from larger untrained ones. Our method is explored and extractsworks by exploring different topologies which are sampled using Gumbel Softmax. The resultingworks are further enhanced using a highly efficient rescaling mechanism that reduces training time and improves performances.
• Score: 9.176056742068813
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large and performant neural networks are often overparameterized and can be drastically reduced in size and complexity thanks to pruning. Pruning is a group of methods, which seeks to remove redundant or unnecessary weights or groups of weights in a network. These techniques allow the creation of lightweight networks, which are particularly critical in embedded or mobile applications. In this paper, we devise an alternative pruning method that allows extracting effective subnetworks from larger untrained ones. Our method is stochastic and extracts subnetworks by exploring different topologies which are sampled using Gumbel Softmax. The latter is also used to train probability distributions which measure the relevance of weights in the sampled topologies. The resulting subnetworks are further enhanced using a highly efficient rescaling mechanism that reduces training time and improves performances. Extensive experiments conducted on CIFAR10 show the outperformance of our subnetwork extraction method against the related work.

Related papers

• Fast Conditional Network Compression Using Bayesian HyperNetworks [54.06346724244786]
  We introduce a conditional compression problem and propose a fast framework for tackling it. The problem is how to quickly compress a pretrained large neural network into optimal smaller networks given target contexts. Our methods can quickly generate compressed networks with significantly smaller sizes than baseline methods.
  arXiv  Detail & Related papers  (2022-05-13T00:28:35Z)
• The Unreasonable Effectiveness of Random Pruning: Return of the Most Naive Baseline for Sparse Training [111.15069968583042]
  Random pruning is arguably the most naive way to attain sparsity in neural networks, but has been deemed uncompetitive by either post-training pruning or sparse training. We empirically demonstrate that sparsely training a randomly pruned network from scratch can match the performance of its dense equivalent. Our results strongly suggest there is larger-than-expected room for sparse training at scale, and the benefits of sparsity might be more universal beyond carefully designed pruning.
  arXiv  Detail & Related papers  (2022-02-05T21:19:41Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Synthesis and Pruning as a Dynamic Compression Strategy for Efficient Deep Neural Networks [1.8275108630751844]
  We propose a novel strategic synthesis algorithm for feedforward networks that draws directly from the brain's behaviours when learning. Unlike existing approaches that advocate random selection, we select highly performing nodes as starting points for new edges. The strategy aims only to produce useful connections and result in a smaller residual network structure.
  arXiv  Detail & Related papers  (2020-11-23T12:30:57Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• Fitting the Search Space of Weight-sharing NAS with Graph Convolutional Networks [100.14670789581811]
  We train a graph convolutional network to fit the performance of sampled sub-networks. With this strategy, we achieve a higher rank correlation coefficient in the selected set of candidates.
  arXiv  Detail & Related papers  (2020-04-17T19:12:39Z)
• DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
  This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
  arXiv  Detail & Related papers  (2020-03-30T17:59:18Z)
• Good Subnetworks Provably Exist: Pruning via Greedy Forward Selection [35.121856435677564]
  We propose a simple greedy selection approach for finding goodworks in deep neural networks. Applying the greedy selection strategy on sufficiently large pre-trained networks guarantees to find smallworks with lower loss than networks directly trained with gradient descent.
  arXiv  Detail & Related papers  (2020-03-03T21:03:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.