Rethinking Class-Discrimination Based CNN Channel Pruning

• URL: http://arxiv.org/abs/2004.14492v1
• Date: Wed, 29 Apr 2020 21:40:23 GMT
• Title: Rethinking Class-Discrimination Based CNN Channel Pruning
• Authors: Yuchen Liu, David Wentzlaff, and S.Y. Kung
• Abstract summary: We study the effectiveness of a broad range of discriminant functions on channel pruning. We develop a FLOP-normalized sensitivity analysis scheme to automate the structural pruning procedure. Our pruned models achieve higher accuracy with less inference cost compared to state-of-the-art results.
• Score: 14.574489739794581
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Channel pruning has received ever-increasing focus on network compression. In particular, class-discrimination based channel pruning has made major headway, as it fits seamlessly with the classification objective of CNNs and provides good explainability. Prior works singly propose and evaluate their discriminant functions, while further study on the effectiveness of the adopted metrics is absent. To this end, we initiate the first study on the effectiveness of a broad range of discriminant functions on channel pruning. Conventional single-variate binary-class statistics like Student's T-Test are also included in our study via an intuitive generalization. The winning metric of our study has a greater ability to select informative channels over other state-of-the-art methods, which is substantiated by our qualitative and quantitative analysis. Moreover, we develop a FLOP-normalized sensitivity analysis scheme to automate the structural pruning procedure. On CIFAR-10, CIFAR-100, and ILSVRC-2012 datasets, our pruned models achieve higher accuracy with less inference cost compared to state-of-the-art results. For example, on ILSVRC-2012, our 44.3% FLOPs-pruned ResNet-50 has only a 0.3% top-1 accuracy drop, which significantly outperforms the state of the art.

Related papers

• Augmenting Unsupervised Reinforcement Learning with Self-Reference [63.68018737038331]
  Humans possess the ability to draw on past experiences explicitly when learning new tasks. We propose the Self-Reference (SR) approach, an add-on module explicitly designed to leverage historical information. Our approach achieves state-of-the-art results in terms of Interquartile Mean (IQM) performance and Optimality Gap reduction on the Unsupervised Reinforcement Learning Benchmark.
  arXiv  Detail & Related papers  (2023-11-16T09:07:34Z)
• Class-Discriminative CNN Compression [10.675326899147802]
  We propose class-discriminative compression (CDC), which injects class discrimination in both pruning and distillation to facilitate the CNNs training goal. CDC is evaluated on CIFAR and ILSVRC 2012, where we consistently outperform the state-of-the-art results.
  arXiv  Detail & Related papers  (2021-10-21T02:54:05Z)
• Test-time Batch Statistics Calibration for Covariate Shift [66.7044675981449]
  We propose to adapt the deep models to the novel environment during inference. We present a general formulation $alpha$-BN to calibrate the batch statistics. We also present a novel loss function to form a unified test time adaptation framework Core.
  arXiv  Detail & Related papers  (2021-10-06T08:45:03Z)
• AIP: Adversarial Iterative Pruning Based on Knowledge Transfer for Convolutional Neural Networks [7.147985297123097]
  convolutional neural networks (CNNs) take a fair amount of computation cost. Current pruning methods can compress CNNs with little performance drop, but when the pruning ratio increases, the accuracy loss is more serious. We propose a novel adversarial iterative pruning method (AIP) for CNNs based on knowledge transfer.
  arXiv  Detail & Related papers  (2021-08-31T02:38:36Z)
• Calibrating Class Activation Maps for Long-Tailed Visual Recognition [60.77124328049557]
  We present two effective modifications of CNNs to improve network learning from long-tailed distribution. First, we present a Class Activation Map (CAMC) module to improve the learning and prediction of network classifiers. Second, we investigate the use of normalized classifiers for representation learning in long-tailed problems.
  arXiv  Detail & Related papers  (2021-08-29T05:45:03Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• BWCP: Probabilistic Learning-to-Prune Channels for ConvNets via Batch Whitening [63.081808698068365]
  This work presents a probabilistic channel pruning method to accelerate Convolutional Neural Networks (CNNs) Previous pruning methods often zero out unimportant channels in training in a deterministic manner, which reduces CNN's learning capacity and results in suboptimal performance. We develop a probability-based pruning algorithm, called batch whitening channel pruning (BWCP), which canally discard unimportant channels by modeling the probability of a channel being activated.
  arXiv  Detail & Related papers  (2021-05-13T17:00:05Z)
• Prune Responsibly [0.913755431537592]
  Irrespective of the specific definition of fairness in a machine learning application, pruning the underlying model affects it. We investigate and document the emergence and exacerbation of undesirable per-class performance imbalances, across tasks and architectures, for almost one million categories considered across over 100K image classification models that undergo a pruning process. We demonstrate the need for transparent reporting, inclusive of bias, fairness, and inclusion metrics, in real-life engineering decision-making around neural network pruning.
  arXiv  Detail & Related papers  (2020-09-10T04:43:11Z)
• Self-Challenging Improves Cross-Domain Generalization [81.99554996975372]
  Convolutional Neural Networks (CNN) conduct image classification by activating dominant features that correlated with labels. We introduce a simple training, Self-Challenging Representation (RSC), that significantly improves the generalization of CNN to the out-of-domain data. RSC iteratively challenges the dominant features activated on the training data, and forces the network to activate remaining features that correlates with labels.
  arXiv  Detail & Related papers  (2020-07-05T21:42:26Z)
• A Feature-map Discriminant Perspective for Pruning Deep Neural Networks [24.062226363823257]
  We present a new mathematical formulation to accurately and efficiently quantify the feature-map discriminativeness. We analyze the theoretical property of DI, specifically the non-decreasing property, that makes DI a valid selection criterion. We propose a DI-based greedy pruning algorithm and structure distillation technique to automatically decide the pruned structure.
  arXiv  Detail & Related papers  (2020-05-28T06:25:22Z)
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks [6.534515590778012]
  Pruning is one of the predominant approaches used for deep network compression. We present a simple-yet-effective gradual channel pruning while training methodology using a novel data-driven metric. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet.
  arXiv  Detail & Related papers  (2020-02-23T17:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.