Tensor Reordering for CNN Compression

• URL: http://arxiv.org/abs/2010.12110v1
• Date: Thu, 22 Oct 2020 23:45:34 GMT
• Title: Tensor Reordering for CNN Compression
• Authors: Matej Ulicny, Vladimir A. Krylov and Rozenn Dahyot
• Abstract summary: We show how parameter redundancy in Convolutional Neural Network (CNN) filters can be effectively reduced by pruning in spectral domain. Our approach is applied to pretrained CNNs and we show that minor additional fine-tuning allows our method to recover the original model performance.
• Score: 7.228285747845778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We show how parameter redundancy in Convolutional Neural Network (CNN) filters can be effectively reduced by pruning in spectral domain. Specifically, the representation extracted via Discrete Cosine Transform (DCT) is more conducive for pruning than the original space. By relying on a combination of weight tensor reshaping and reordering we achieve high levels of layer compression with just minor accuracy loss. Our approach is applied to compress pretrained CNNs and we show that minor additional fine-tuning allows our method to recover the original model performance after a significant parameter reduction. We validate our approach on ResNet-50 and MobileNet-V2 architectures for ImageNet classification task.

Related papers

• Convolutional Neural Network Compression via Dynamic Parameter Rank Pruning [4.7027290803102675]
  We propose an efficient training method for CNN compression via dynamic parameter rank pruning. Our experiments show that the proposed method can yield substantial storage savings while maintaining or even enhancing classification performance.
  arXiv  Detail & Related papers  (2024-01-15T23:52:35Z)
• Compression with Bayesian Implicit Neural Representations [16.593537431810237]
  We propose overfitting variational neural networks to the data and compressing an approximate posterior weight sample using relative entropy coding instead of quantizing and entropy coding it. Experiments show that our method achieves strong performance on image and audio compression while retaining simplicity.
  arXiv  Detail & Related papers  (2023-05-30T16:29:52Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• Approximating Continuous Convolutions for Deep Network Compression [11.566258236184964]
  We present ApproxConv, a novel method for compressing the layers of a convolutional neural network. We show that our method is able to compress existing deep network models by half whilst losing only 1.86% accuracy.
  arXiv  Detail & Related papers  (2022-10-17T11:41:26Z)
• Reducing Redundancy in the Bottleneck Representation of the Autoencoders [98.78384185493624]
  Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks. We propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST.
  arXiv  Detail & Related papers  (2022-02-09T18:48:02Z)
• Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization [1.8579693774597708]
  We propose a model reduction method to compress the pre-trained networks using low-rank tensor decomposition. A new regularization method, called funnel function, is proposed to suppress the unimportant factors during the compression. For ResNet18 with ImageNet2012, our reduced model can reach more than twi times speed up in terms of GMAC with merely 0.7% Top-1 accuracy drop.
  arXiv  Detail & Related papers  (2021-12-07T13:41:51Z)
• Toward Compact Parameter Representations for Architecture-Agnostic Neural Network Compression [26.501979992447605]
  This paper investigates compression from the perspective of compactly representing and storing trained parameters. We leverage additive quantization, an extreme lossy compression method invented for image descriptors, to compactly represent the parameters. We conduct experiments on MobileNet-v2, VGG-11, ResNet-50, Feature Pyramid Networks, and pruned DNNs trained for classification, detection, and segmentation tasks.
  arXiv  Detail & Related papers  (2021-11-19T17:03:11Z)
• Rate Distortion Characteristic Modeling for Neural Image Compression [59.25700168404325]
  End-to-end optimization capability offers neural image compression (NIC) superior lossy compression performance. distinct models are required to be trained to reach different points in the rate-distortion (R-D) space. We make efforts to formulate the essential mathematical functions to describe the R-D behavior of NIC using deep network and statistical modeling.
  arXiv  Detail & Related papers  (2021-06-24T12:23:05Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)
• ResRep: Lossless CNN Pruning via Decoupling Remembering and Forgetting [105.97936163854693]
  We propose ResRep, which slims down a CNN by reducing the width (number of output channels) of convolutional layers. Inspired by the neurobiology research about the independence of remembering and forgetting, we propose to re- parameterize a CNN into the remembering parts and forgetting parts. We equivalently merge the remembering and forgetting parts into the original architecture with narrower layers.
  arXiv  Detail & Related papers  (2020-07-07T07:56:45Z)
• Perceptually Optimizing Deep Image Compression [53.705543593594285]
  Mean squared error (MSE) and $ell_p$ norms have largely dominated the measurement of loss in neural networks. We propose a different proxy approach to optimize image analysis networks against quantitative perceptual models.
  arXiv  Detail & Related papers  (2020-07-03T14:33:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.