Class-Discriminative CNN Compression

• URL: http://arxiv.org/abs/2110.10864v1
• Date: Thu, 21 Oct 2021 02:54:05 GMT
• Title: Class-Discriminative CNN Compression
• Authors: Yuchen Liu, David Wentzlaff, S.Y. Kung
• Abstract summary: We propose class-discriminative compression (CDC), which injects class discrimination in both pruning and distillation to facilitate the CNNs training goal. CDC is evaluated on CIFAR and ILSVRC 2012, where we consistently outperform the state-of-the-art results.
• Score: 10.675326899147802
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Compressing convolutional neural networks (CNNs) by pruning and distillation has received ever-increasing focus in the community. In particular, designing a class-discrimination based approach would be desired as it fits seamlessly with the CNNs training objective. In this paper, we propose class-discriminative compression (CDC), which injects class discrimination in both pruning and distillation to facilitate the CNNs training goal. We first study the effectiveness of a group of discriminant functions for channel pruning, where we include well-known single-variate binary-class statistics like Student's T-Test in our study via an intuitive generalization. We then propose a novel layer-adaptive hierarchical pruning approach, where we use a coarse class discrimination scheme for early layers and a fine one for later layers. This method naturally accords with the fact that CNNs process coarse semantics in the early layers and extract fine concepts at the later. Moreover, we leverage discriminant component analysis (DCA) to distill knowledge of intermediate representations in a subspace with rich discriminative information, which enhances hidden layers' linear separability and classification accuracy of the student. Combining pruning and distillation, CDC is evaluated on CIFAR and ILSVRC 2012, where we consistently outperform the state-of-the-art results.

Related papers

• Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization [64.36097398869774]
  Semi-supervised learning (SSL) has been an active research topic for large-scale 3D scene understanding. The existing SSL-based methods suffer from severe training bias due to class imbalance and long-tail distributions of the point cloud data. We introduce a new decoupling optimization framework, which disentangles feature representation learning and classifier in an alternative optimization manner to shift the bias decision boundary effectively.
  arXiv  Detail & Related papers  (2024-01-13T04:16:40Z)
• PICNN: A Pathway towards Interpretable Convolutional Neural Networks [12.31424771480963]
  We introduce a novel pathway to alleviate the entanglement between filters and image classes. We use the Bernoulli sampling to generate the filter-cluster assignment matrix from a learnable filter-class correspondence matrix. We evaluate the effectiveness of our method on ten widely used network architectures.
  arXiv  Detail & Related papers  (2023-12-19T11:36:03Z)
• Bi-discriminator Domain Adversarial Neural Networks with Class-Level Gradient Alignment [87.8301166955305]
  We propose a novel bi-discriminator domain adversarial neural network with class-level gradient alignment. BACG resorts to gradient signals and second-order probability estimation for better alignment of domain distributions. In addition, inspired by contrastive learning, we develop a memory bank-based variant, i.e. Fast-BACG, which can greatly shorten the training process.
  arXiv  Detail & Related papers  (2023-10-21T09:53:17Z)
• Feature Selection using Sparse Adaptive Bottleneck Centroid-Encoder [1.2487990897680423]
  We introduce a novel nonlinear model, Sparse Adaptive Bottleneckid-Encoder (SABCE), for determining the features that discriminate between two or more classes. The algorithm is applied to various real-world data sets, including high-dimensional biological, image, speech, and accelerometer sensor data.
  arXiv  Detail & Related papers  (2023-06-07T21:37:21Z)
• Understanding Imbalanced Semantic Segmentation Through Neural Collapse [81.89121711426951]
  We show that semantic segmentation naturally brings contextual correlation and imbalanced distribution among classes. We introduce a regularizer on feature centers to encourage the network to learn features closer to the appealing structure. Our method ranks 1st and sets a new record on the ScanNet200 test leaderboard.
  arXiv  Detail & Related papers  (2023-01-03T13:51:51Z)
• Do We Really Need a Learnable Classifier at the End of Deep Neural Network? [118.18554882199676]
  We study the potential of learning a neural network for classification with the classifier randomly as an ETF and fixed during training. Our experimental results show that our method is able to achieve similar performances on image classification for balanced datasets.
  arXiv  Detail & Related papers  (2022-03-17T04:34:28Z)
• Fairness via Representation Neutralization [60.90373932844308]
  We propose a new mitigation technique, namely, Representation Neutralization for Fairness (RNF) RNF achieves that fairness by debiasing only the task-specific classification head of DNN models. Experimental results over several benchmark datasets demonstrate our RNF framework to effectively reduce discrimination of DNN models.
  arXiv  Detail & Related papers  (2021-06-23T22:26:29Z)
• No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data [78.69828864672978]
  A central challenge in training classification models in the real-world federated system is learning with non-IID data. We propose a novel and simple algorithm called Virtual Representations (CCVR), which adjusts the classifier using virtual representations sampled from an approximated ssian mixture model. Experimental results demonstrate that CCVR state-of-the-art performance on popular federated learning benchmarks including CIFAR-10, CIFAR-100, and CINIC-10.
  arXiv  Detail & Related papers  (2021-06-09T12:02:29Z)
• Food Classification with Convolutional Neural Networks and Multi-Class Linear Discernment Analysis [0.0]
  Linear discriminant analysis (LDA) can be implemented in a multi-class classification method to increase separability of class features. CNN is superior to LDA for image classification and why LDA should not be left out of the races for image classification.
  arXiv  Detail & Related papers  (2020-12-06T03:28:58Z)
• A Feature-map Discriminant Perspective for Pruning Deep Neural Networks [24.062226363823257]
  We present a new mathematical formulation to accurately and efficiently quantify the feature-map discriminativeness. We analyze the theoretical property of DI, specifically the non-decreasing property, that makes DI a valid selection criterion. We propose a DI-based greedy pruning algorithm and structure distillation technique to automatically decide the pruned structure.
  arXiv  Detail & Related papers  (2020-05-28T06:25:22Z)
• Rethinking Class-Discrimination Based CNN Channel Pruning [14.574489739794581]
  We study the effectiveness of a broad range of discriminant functions on channel pruning. We develop a FLOP-normalized sensitivity analysis scheme to automate the structural pruning procedure. Our pruned models achieve higher accuracy with less inference cost compared to state-of-the-art results.
  arXiv  Detail & Related papers  (2020-04-29T21:40:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.