Retrain or not retrain? -- efficient pruning methods of deep CNN networks

• URL: http://arxiv.org/abs/2002.07051v1
• Date: Wed, 12 Feb 2020 23:24:28 GMT
• Title: Retrain or not retrain? -- efficient pruning methods of deep CNN networks
• Authors: Marcin Pietron and Maciej Wielgosz
• Abstract summary: Convolutional neural networks (CNN) play a major role in image processing tasks like image classification, object detection, semantic segmentation. Very often CNN networks have from several to hundred stacked layers with several megabytes of weights. One of the possible methods to reduce complexity and memory footprint is pruning.
• Score: 0.30458514384586394
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Convolutional neural networks (CNN) play a major role in image processing tasks like image classification, object detection, semantic segmentation. Very often CNN networks have from several to hundred stacked layers with several megabytes of weights. One of the possible methods to reduce complexity and memory footprint is pruning. Pruning is a process of removing weights which connect neurons from two adjacent layers in the network. The process of finding near optimal solution with specified drop in accuracy can be more sophisticated when DL model has higher number of convolutional layers. In the paper few approaches based on retraining and no retraining are described and compared together.

Related papers

• Model Parallel Training and Transfer Learning for Convolutional Neural Networks by Domain Decomposition [0.0]
  Deep convolutional neural networks (CNNs) have been shown to be very successful in a wide range of image processing applications. Due to their increasing number of model parameters and an increasing availability of large amounts of training data, parallelization strategies to efficiently train complex CNNs are necessary.
  arXiv  Detail & Related papers  (2024-08-26T17:35:01Z)
• Training Convolutional Neural Networks with the Forward-Forward algorithm [1.74440662023704]
  Forward Forward (FF) algorithm has up to now only been used in fully connected networks. We show how the FF paradigm can be extended to CNNs. Our FF-trained CNN, featuring a novel spatially-extended labeling technique, achieves a classification accuracy of 99.16% on the MNIST hand-written digits dataset.
  arXiv  Detail & Related papers  (2023-12-22T18:56:35Z)
• Training Your Sparse Neural Network Better with Any Mask [106.134361318518]
  Pruning large neural networks to create high-quality, independently trainable sparse masks is desirable. In this paper we demonstrate an alternative opportunity: one can customize the sparse training techniques to deviate from the default dense network training protocols. Our new sparse training recipe is generally applicable to improving training from scratch with various sparse masks.
  arXiv  Detail & Related papers  (2022-06-26T00:37:33Z)
• Recursive Least Squares for Training and Pruning Convolutional Neural Networks [27.089496826735672]
  Convolutional neural networks (CNNs) have succeeded in many practical applications. High computation and storage requirements make them difficult to deploy on resource-constrained devices. We propose a novel algorithm for training and pruning CNNs.
  arXiv  Detail & Related papers  (2022-01-13T07:14:08Z)
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods [73.03533268740605]
  We propose a novel scheduled grow-and-prune (GaP) methodology without pre-training the dense models. Experiments have shown that such models can match or beat the quality of highly optimized dense models at 80% sparsity on a variety of tasks.
  arXiv  Detail & Related papers  (2021-06-18T01:03:13Z)
• ResMLP: Feedforward networks for image classification with data-efficient training [73.26364887378597]
  We present ResMLP, an architecture built entirely upon multi-layer perceptrons for image classification. We will share our code based on the Timm library and pre-trained models.
  arXiv  Detail & Related papers  (2021-05-07T17:31:44Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Local Critic Training for Model-Parallel Learning of Deep Neural Networks [94.69202357137452]
  We propose a novel model-parallel learning method, called local critic training. We show that the proposed approach successfully decouples the update process of the layer groups for both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) We also show that trained networks by the proposed method can be used for structural optimization.
  arXiv  Detail & Related papers  (2021-02-03T09:30:45Z)
• An End-To-End-Trainable Iterative Network Architecture for Accelerated Radial Multi-Coil 2D Cine MR Image Reconstruction [4.233498905999929]
  We propose a CNN-architecture for image reconstruction of accelerated 2D radial cine MRI with multiple receiver coils. We investigate the proposed training-strategy and compare our method to other well-known reconstruction techniques with learned and non-learned regularization methods.
  arXiv  Detail & Related papers  (2021-02-01T11:42:04Z)
• ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
  We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks. Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
  arXiv  Detail & Related papers  (2020-06-28T23:09:27Z)
• Exploiting Weight Redundancy in CNNs: Beyond Pruning and Quantization [0.2538209532048866]
  Pruning and quantization are proven methods for improving the performance and storage efficiency of convolutional neural networks (CNNs) We identify another form of redundancy in CNN weight tensors, in the form of repeated patterns of similar values.
  arXiv  Detail & Related papers  (2020-06-22T01:54:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.