Interpretations Steered Network Pruning via Amortized Inferred Saliency Maps

• URL: http://arxiv.org/abs/2209.02869v1
• Date: Wed, 7 Sep 2022 01:12:11 GMT
• Title: Interpretations Steered Network Pruning via Amortized Inferred Saliency Maps
• Authors: Alireza Ganjdanesh, Shangqian Gao and Heng Huang
• Abstract summary: Convolutional Neural Networks (CNNs) compression is crucial to deploying these models in edge devices with limited resources. We propose to address the channel pruning problem from a novel perspective by leveraging the interpretations of a model to steer the pruning process. We tackle this challenge by introducing a selector model that predicts real-time smooth saliency masks for pruned models.
• Score: 85.49020931411825
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Convolutional Neural Networks (CNNs) compression is crucial to deploying these models in edge devices with limited resources. Existing channel pruning algorithms for CNNs have achieved plenty of success on complex models. They approach the pruning problem from various perspectives and use different metrics to guide the pruning process. However, these metrics mainly focus on the model's `outputs' or `weights' and neglect its `interpretations' information. To fill in this gap, we propose to address the channel pruning problem from a novel perspective by leveraging the interpretations of a model to steer the pruning process, thereby utilizing information from both inputs and outputs of the model. However, existing interpretation methods cannot get deployed to achieve our goal as either they are inefficient for pruning or may predict non-coherent explanations. We tackle this challenge by introducing a selector model that predicts real-time smooth saliency masks for pruned models. We parameterize the distribution of explanatory masks by Radial Basis Function (RBF)-like functions to incorporate geometric prior of natural images in our selector model's inductive bias. Thus, we can obtain compact representations of explanations to reduce the computational costs of our pruning method. We leverage our selector model to steer the network pruning by maximizing the similarity of explanatory representations for the pruned and original models. Extensive experiments on CIFAR-10 and ImageNet benchmark datasets demonstrate the efficacy of our proposed method. Our implementations are available at \url{https://github.com/Alii-Ganjj/InterpretationsSteeredPruning}

Related papers

• RL-Pruner: Structured Pruning Using Reinforcement Learning for CNN Compression and Acceleration [0.0]
  We propose RL-Pruner, which uses reinforcement learning to learn the optimal pruning distribution. RL-Pruner can automatically extract dependencies between filters in the input model and perform pruning, without requiring model-specific pruning implementations.
  arXiv  Detail & Related papers  (2024-11-10T13:35:10Z)
• FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models [56.71672127740099]
  We focus on the task of image segmentation, which is traditionally solved by training models on closed-vocabulary datasets. We leverage different and relatively small-sized, open-source foundation models for zero-shot open-vocabulary segmentation. Our approach (dubbed FreeSeg-Diff), which does not rely on any training, outperforms many training-based approaches on both Pascal VOC and COCO datasets.
  arXiv  Detail & Related papers  (2024-03-29T10:38:25Z)
• Diffusion models for probabilistic programming [56.47577824219207]
  Diffusion Model Variational Inference (DMVI) is a novel method for automated approximate inference in probabilistic programming languages (PPLs) DMVI is easy to implement, allows hassle-free inference in PPLs without the drawbacks of, e.g., variational inference using normalizing flows, and does not make any constraints on the underlying neural network model.
  arXiv  Detail & Related papers  (2023-11-01T12:17:05Z)
• Studying How to Efficiently and Effectively Guide Models with Explanations [52.498055901649025]
  'Model guidance' is the idea of regularizing the models' explanations to ensure that they are "right for the right reasons" We conduct an in-depth evaluation across various loss functions, attribution methods, models, and 'guidance depths' on the PASCAL VOC 2007 and MS COCO 2014 datasets. Specifically, we guide the models via bounding box annotations, which are much cheaper to obtain than the commonly used segmentation masks.
  arXiv  Detail & Related papers  (2023-03-21T15:34:50Z)
• Efficient Stein Variational Inference for Reliable Distribution-lossless Network Pruning [23.22021752821507]
  We propose a novel distribution-lossless pruning method, named vanillaP, to theoretically find the pruned lottery within Bayesian treatment. Our method can obtain sparser networks with great performance while providing quantified reliability for the pruned model.
  arXiv  Detail & Related papers  (2022-12-07T09:31:47Z)
• A Fair Loss Function for Network Pruning [70.35230425589592]
  We introduce the performance weighted loss function, a simple modified cross-entropy loss function that can be used to limit the introduction of biases during pruning. Experiments using the CelebA, Fitzpatrick17k and CIFAR-10 datasets demonstrate that the proposed method is a simple and effective tool.
  arXiv  Detail & Related papers  (2022-11-18T15:17:28Z)
• Towards Accurate Subgraph Similarity Computation via Neural Graph Pruning [22.307526272085024]
  In this work, we convert graph pruning to a problem of node relabeling and then relax it to a differentiable problem. Based on this idea, we further design a novel neural network to approximate a type of subgraph distance: the subgraph edit distance (SED) In the design of the model, we propose an attention mechanism to leverage the information about the query graph and guide the pruning of the target graph.
  arXiv  Detail & Related papers  (2022-10-19T15:16:28Z)
• Paoding: Supervised Robustness-preserving Data-free Neural Network Pruning [3.6953655494795776]
  We study the neural network pruning in the emphdata-free context. We replace the traditional aggressive one-shot strategy with a conservative one that treats the pruning as a progressive process. Our method is implemented as a Python package named textscPaoding and evaluated with a series of experiments on diverse neural network models.
  arXiv  Detail & Related papers  (2022-04-02T07:09:17Z)
• Dynamic Model Pruning with Feedback [64.019079257231]
  We propose a novel model compression method that generates a sparse trained model without additional overhead. We evaluate our method on CIFAR-10 and ImageNet, and show that the obtained sparse models can reach the state-of-the-art performance of dense models.
  arXiv  Detail & Related papers  (2020-06-12T15:07:08Z)
• Modeling of Pruning Techniques for Deep Neural Networks Simplification [8.75217589103206]
  Convolutional Neural Networks (CNNs) suffer from different issues, such as computational complexity and the number of parameters. In recent years pruning techniques are employed to reduce the number of operations and model size in CNNs. Various techniques and tricks accompany pruning methods, and there is not a unifying framework to model all the pruning methods.
  arXiv  Detail & Related papers  (2020-01-13T04:51:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.