Compressing Deep CNNs using Basis Representation and Spectral Fine-tuning

• URL: http://arxiv.org/abs/2105.10436v1
• Date: Fri, 21 May 2021 16:14:26 GMT
• Title: Compressing Deep CNNs using Basis Representation and Spectral Fine-tuning
• Authors: Muhammad Tayyab, Fahad Ahmad Khan, Abhijit Mahalanobis
• Abstract summary: We propose an efficient and straightforward method for compressing deep convolutional neural networks (CNNs) Specifically, any spatial convolution layer of the CNN can be replaced by two successive convolution layers. We fine-tune both the basis and the filter representation to directly mitigate any performance loss due to the truncation.
• Score: 2.578242050187029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose an efficient and straightforward method for compressing deep convolutional neural networks (CNNs) that uses basis filters to represent the convolutional layers, and optimizes the performance of the compressed network directly in the basis space. Specifically, any spatial convolution layer of the CNN can be replaced by two successive convolution layers: the first is a set of three-dimensional orthonormal basis filters, followed by a layer of one-dimensional filters that represents the original spatial filters in the basis space. We jointly fine-tune both the basis and the filter representation to directly mitigate any performance loss due to the truncation. Generality of the proposed approach is demonstrated by applying it to several well known deep CNN architectures and data sets for image classification and object detection. We also present the execution time and power usage at different compression levels on the Xavier Jetson AGX processor.

Related papers

• DNN-Based Topology Optimisation: Spatial Invariance and Neural Tangent Kernel [7.106986689736828]
  We study the SIMP method with a density field generated by a fully-connected neural network, taking the coordinates as inputs. We show that the use of DNNs leads to a filtering effect similar to traditional filtering techniques for SIMP, with a filter described by the Neural Tangent Kernel (NTK)
  arXiv  Detail & Related papers  (2021-06-10T12:49:55Z)
• Resolution learning in deep convolutional networks using scale-space theory [31.275270391367425]
  Resolution in deep convolutional neural networks (CNNs) is typically bounded by the receptive field size through filter sizes, and subsampling layers or strided convolutions on feature maps. We propose to do away with hard-coded resolution hyper- parameters and aim to learn the appropriate resolution from data. We use scale-space theory to obtain a self-similar parametrization of filters and make use of the N-Jet: a truncated Taylor series to approximate a filter by a learned combination of Gaussian derivative filters.
  arXiv  Detail & Related papers  (2021-06-07T08:23:02Z)
• Unsharp Mask Guided Filtering [53.14430987860308]
  The goal of this paper is guided image filtering, which emphasizes the importance of structure transfer during filtering. We propose a new and simplified formulation of the guided filter inspired by unsharp masking. Our formulation enjoys a filtering prior to a low-pass filter and enables explicit structure transfer by estimating a single coefficient.
  arXiv  Detail & Related papers  (2021-06-02T19:15:34Z)
• Decoupled Dynamic Filter Networks [85.38058820176047]
  We propose the Decoupled Dynamic Filter (DDF) that can simultaneously tackle both of these shortcomings. Inspired by recent advances in attention, DDF decouples a depth-wise dynamic filter into spatial and channel dynamic filters. We observe a significant boost in performance when replacing standard convolution with DDF in classification networks.
  arXiv  Detail & Related papers  (2021-04-29T04:55:33Z)
• Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks [70.0243910593064]
  Key to success of vector quantization is deciding which parameter groups should be compressed together. In this paper we make the observation that the weights of two adjacent layers can be permuted while expressing the same function. We then establish a connection to rate-distortion theory and search for permutations that result in networks that are easier to compress.
  arXiv  Detail & Related papers  (2020-10-29T15:47:26Z)
• Tensor Reordering for CNN Compression [7.228285747845778]
  We show how parameter redundancy in Convolutional Neural Network (CNN) filters can be effectively reduced by pruning in spectral domain. Our approach is applied to pretrained CNNs and we show that minor additional fine-tuning allows our method to recover the original model performance.
  arXiv  Detail & Related papers  (2020-10-22T23:45:34Z)
• Unrolling of Deep Graph Total Variation for Image Denoising [106.93258903150702]
  In this paper, we combine classical graph signal filtering with deep feature learning into a competitive hybrid design. We employ interpretable analytical low-pass graph filters and employ 80% fewer network parameters than state-of-the-art DL denoising scheme DnCNN.
  arXiv  Detail & Related papers  (2020-10-21T20:04:22Z)
• Compressing Deep Convolutional Neural Networks by Stacking Low-dimensional Binary Convolution Filters [15.66437882635872]
  Deep Convolutional Neural Networks (CNN) have been successfully applied to many real-life problems. Huge memory cost of deep CNN models poses a great challenge of deploying them on memory-constrained devices. We propose a novel method to compress deep CNN model by stacking low-dimensional binary convolution filters.
  arXiv  Detail & Related papers  (2020-10-06T14:49:22Z)
• Implicit Convex Regularizers of CNN Architectures: Convex Optimization of Two- and Three-Layer Networks in Polynomial Time [70.15611146583068]
  We study training of Convolutional Neural Networks (CNNs) with ReLU activations. We introduce exact convex optimization with a complexity with respect to the number of data samples, the number of neurons, and data dimension.
  arXiv  Detail & Related papers  (2020-06-26T04:47:20Z)
• Real-Time High-Performance Semantic Image Segmentation of Urban Street Scenes [98.65457534223539]
  We propose a real-time high-performance DCNN-based method for robust semantic segmentation of urban street scenes. The proposed method achieves the accuracy of 73.6% and 68.0% mean Intersection over Union (mIoU) with the inference speed of 51.0 fps and 39.3 fps.
  arXiv  Detail & Related papers  (2020-03-11T08:45:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.