Accelerating Convolutional Neural Network Pruning via Spatial Aura Entropy

• URL: http://arxiv.org/abs/2312.04926v1
• Date: Fri, 8 Dec 2023 09:43:49 GMT
• Title: Accelerating Convolutional Neural Network Pruning via Spatial Aura Entropy
• Authors: Bogdan Musat, Razvan Andonie
• Abstract summary: pruning is a popular technique to reduce the computational complexity and memory footprint of Convolutional Neural Network (CNN) models. Existing methods for MI computation suffer from high computational cost and sensitivity to noise, leading to suboptimal pruning performance. We propose a novel method to improve MI computation for CNN pruning, using the spatial aura entropy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, pruning has emerged as a popular technique to reduce the computational complexity and memory footprint of Convolutional Neural Network (CNN) models. Mutual Information (MI) has been widely used as a criterion for identifying unimportant filters to prune. However, existing methods for MI computation suffer from high computational cost and sensitivity to noise, leading to suboptimal pruning performance. We propose a novel method to improve MI computation for CNN pruning, using the spatial aura entropy. The spatial aura entropy is useful for evaluating the heterogeneity in the distribution of the neural activations over a neighborhood, providing information about local features. Our method effectively improves the MI computation for CNN pruning, leading to more robust and efficient pruning. Experimental results on the CIFAR-10 benchmark dataset demonstrate the superiority of our approach in terms of pruning performance and computational efficiency.

Related papers

• Towards Generalized Entropic Sparsification for Convolutional Neural Networks [0.0]
  Convolutional neural networks (CNNs) are reported to be overparametrized. Here, we introduce a layer-by-layer data-driven pruning method based on the mathematical idea aiming at a computationally-scalable entropic relaxation of the pruning problem. The sparse subnetwork is found from the pre-trained (full) CNN using the network entropy minimization as a sparsity constraint.
  arXiv  Detail & Related papers  (2024-04-06T21:33:39Z)
• Deep Multi-Threshold Spiking-UNet for Image Processing [51.88730892920031]
  This paper introduces the novel concept of Spiking-UNet for image processing, which combines the power of Spiking Neural Networks (SNNs) with the U-Net architecture. To achieve an efficient Spiking-UNet, we face two primary challenges: ensuring high-fidelity information propagation through the network via spikes and formulating an effective training strategy. Experimental results show that, on image segmentation and denoising, our Spiking-UNet achieves comparable performance to its non-spiking counterpart.
  arXiv  Detail & Related papers  (2023-07-20T16:00:19Z)
• Distributed Neural Representation for Reactive in situ Visualization [23.80657290203846]
  Implicit neural representations (INRs) have emerged as a powerful tool for compressing large-scale volume data. We develop a distributed neural representation and optimize it for in situ visualization. Our technique eliminates data exchanges between processes, achieving state-of-the-art compression speed, quality and ratios.
  arXiv  Detail & Related papers  (2023-03-28T03:55:47Z)
• Spatio-temporal point processes with deep non-stationary kernels [18.10670233156497]
  We develop a new deep non-stationary influence kernel that can model non-stationary-temporal point processes. The main idea is to approximate the influence kernel with a novel and general low-rank decomposition. We also take a new approach to maintain the non-negativity constraint of the conditional intensity by introducing a log-barrier penalty.
  arXiv  Detail & Related papers  (2022-11-21T04:49:39Z)
• Efficient Dataset Distillation Using Random Feature Approximation [109.07737733329019]
  We propose a novel algorithm that uses a random feature approximation (RFA) of the Neural Network Gaussian Process (NNGP) kernel. Our algorithm provides at least a 100-fold speedup over KIP and can run on a single GPU. Our new method, termed an RFA Distillation (RFAD), performs competitively with KIP and other dataset condensation algorithms in accuracy over a range of large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-21T15:56:13Z)
• Improved Algorithms for Neural Active Learning [74.89097665112621]
  We improve the theoretical and empirical performance of neural-network(NN)-based active learning algorithms for the non-parametric streaming setting. We introduce two regret metrics by minimizing the population loss that are more suitable in active learning than the one used in state-of-the-art (SOTA) related work.
  arXiv  Detail & Related papers  (2022-10-02T05:03:38Z)
• Efficient Cluster-Based k-Nearest-Neighbor Machine Translation [65.69742565855395]
  k-Nearest-Neighbor Machine Translation (kNN-MT) has been recently proposed as a non-parametric solution for domain adaptation in neural machine translation (NMT)
  arXiv  Detail & Related papers  (2022-04-13T05:46:31Z)
• Convolutional generative adversarial imputation networks for spatio-temporal missing data in storm surge simulations [86.5302150777089]
  Generative Adversarial Imputation Nets (GANs) and GAN-based techniques have attracted attention as unsupervised machine learning methods. We name our proposed method as Con Conval Generative Adversarial Imputation Nets (Conv-GAIN)
  arXiv  Detail & Related papers  (2021-11-03T03:50:48Z)
• CONet: Channel Optimization for Convolutional Neural Networks [33.58529066005248]
  We study channel size optimization in convolutional neural networks (CNN) We introduce an efficient dynamic scaling algorithm -- CONet -- that automatically optimize channel sizes across network layers for a given CNN. We conduct experiments on CIFAR10/100 and ImageNet datasets and show that CONet can find efficient and accurate architectures searched in ResNet, DARTS, and DARTS+ spaces.
  arXiv  Detail & Related papers  (2021-08-15T21:48:25Z)
• Random Features for the Neural Tangent Kernel [57.132634274795066]
  We propose an efficient feature map construction of the Neural Tangent Kernel (NTK) of fully-connected ReLU network. We show that dimension of the resulting features is much smaller than other baseline feature map constructions to achieve comparable error bounds both in theory and practice.
  arXiv  Detail & Related papers  (2021-04-03T09:08:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.