RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks

• URL: http://arxiv.org/abs/2105.14797v1
• Date: Mon, 31 May 2021 08:44:14 GMT
• Title: RED : Looking for Redundancies for Data-Free Structured Compression of Deep Neural Networks
• Authors: Edouard Yvinec, Arnaud Dapogny, Matthieu Cord and Kevin Bailly
• Abstract summary: We present RED, a data-free structured, unified approach to tackle structured pruning. First, we propose a novel adaptive hashing of the scalar weight distribution densities to increase the number of identical neurons. Second, we prune the network by merging redundant neurons based on their relative similarities, as defined by their distance. Third, we propose a novel uneven depthwise separation technique to further prune convolutional layers.
• Score: 36.027765880474526
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Neural Networks (DNNs) are ubiquitous in today's computer vision land-scape, despite involving considerable computational costs. The mainstream approaches for runtime acceleration consist in pruning connections (unstructured pruning) or, better, filters (structured pruning), both often requiring data to re-train the model. In this paper, we present RED, a data-free structured, unified approach to tackle structured pruning. First, we propose a novel adaptive hashing of the scalar DNN weight distribution densities to increase the number of identical neurons represented by their weight vectors. Second, we prune the network by merging redundant neurons based on their relative similarities, as defined by their distance. Third, we propose a novel uneven depthwise separation technique to further prune convolutional layers. We demonstrate through a large variety of benchmarks that RED largely outperforms other data-free pruning methods, often reaching performance similar to unconstrained, data-driven methods.

Related papers

• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Structured Pruning for Deep Convolutional Neural Networks: A survey [2.811264250666485]
  Pruning neural networks has thus gained interest since it effectively lowers storage and computational costs. This article surveys the recent progress towards structured pruning of deep CNNs. We summarize and compare the state-of-the-art structured pruning techniques with respect to filter ranking methods, regularization methods, dynamic execution, neural architecture search, the lottery ticket hypothesis, and the applications of pruning.
  arXiv  Detail & Related papers  (2023-03-01T15:12:55Z)
• Boosting Pruned Networks with Linear Over-parameterization [8.796518772724955]
  Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. We propose a novel method that first linearly over- parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters.
  arXiv  Detail & Related papers  (2022-04-25T05:30:26Z)
• RED++ : Data-Free Pruning of Deep Neural Networks via Input Splitting and Output Merging [36.027765880474526]
  Pruning Deep Neural Networks (DNNs) is a prominent field of study in the goal of inference runtime acceleration. We introduce a novel data-free pruning protocol RED++. We study the theoretical and empirical guarantees on the preservation of the accuracy from the hashing.
  arXiv  Detail & Related papers  (2021-09-30T09:31:11Z)
• Group Fisher Pruning for Practical Network Compression [58.25776612812883]
  We present a general channel pruning approach that can be applied to various complicated structures. We derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels. Our method can be used to prune any structures including those with coupled channels.
  arXiv  Detail & Related papers  (2021-08-02T08:21:44Z)
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
  Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
  arXiv  Detail & Related papers  (2021-02-08T05:55:47Z)
• Neural Pruning via Growing Regularization [82.9322109208353]
  We extend regularization to tackle two central problems of pruning: pruning schedule and weight importance scoring. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains. The proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning.
  arXiv  Detail & Related papers  (2020-12-16T20:16:28Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• A Framework for Neural Network Pruning Using Gibbs Distributions [34.0576955010317]
  Gibbs pruning is a novel framework for expressing and designing neural network pruning methods. It can train and prune a network simultaneously in such a way that the learned weights and pruning mask are well-adapted for each other. We achieve a new state-of-the-art result for pruning ResNet-56 with the CIFAR-10 dataset.
  arXiv  Detail & Related papers  (2020-06-08T23:04:53Z)
• Large-Scale Gradient-Free Deep Learning with Recursive Local Representation Alignment [84.57874289554839]
  Training deep neural networks on large-scale datasets requires significant hardware resources. Backpropagation, the workhorse for training these networks, is an inherently sequential process that is difficult to parallelize. We propose a neuro-biologically-plausible alternative to backprop that can be used to train deep networks.
  arXiv  Detail & Related papers  (2020-02-10T16:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.