Soft Masking for Cost-Constrained Channel Pruning
- URL: http://arxiv.org/abs/2211.02206v1
- Date: Fri, 4 Nov 2022 01:28:45 GMT
- Title: Soft Masking for Cost-Constrained Channel Pruning
- Authors: Ryan Humble, Maying Shen, Jorge Albericio Latorre, Eric Darve1, Jose
M. Alvarez
- Abstract summary: Structured channel pruning has been shown to significantly accelerate inference time for convolution neural networks (CNNs) on modern hardware.
Recent works permanently zero these channels during training, which we observe to significantly hamper final accuracy.
We propose Soft Masking for cost-constrained Channel Pruning (SMCP) to allow pruned channels to adaptively return to the network.
- Score: 17.138115344464513
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Structured channel pruning has been shown to significantly accelerate
inference time for convolution neural networks (CNNs) on modern hardware, with
a relatively minor loss of network accuracy. Recent works permanently zero
these channels during training, which we observe to significantly hamper final
accuracy, particularly as the fraction of the network being pruned increases.
We propose Soft Masking for cost-constrained Channel Pruning (SMCP) to allow
pruned channels to adaptively return to the network while simultaneously
pruning towards a target cost constraint. By adding a soft mask
re-parameterization of the weights and channel pruning from the perspective of
removing input channels, we allow gradient updates to previously pruned
channels and the opportunity for the channels to later return to the network.
We then formulate input channel pruning as a global resource allocation
problem. Our method outperforms prior works on both the ImageNet classification
and PASCAL VOC detection datasets.
Related papers
- TBSN: Transformer-Based Blind-Spot Network for Self-Supervised Image Denoising [94.09442506816724]
Blind-spot networks (BSN) have been prevalent network architectures in self-supervised image denoising (SSID)
We present a transformer-based blind-spot network (TBSN) by analyzing and redesigning the transformer operators that meet the blind-spot requirement.
For spatial self-attention, an elaborate mask is applied to the attention matrix to restrict its receptive field, thus mimicking the dilated convolution.
For channel self-attention, we observe that it may leak the blind-spot information when the channel number is greater than spatial size in the deep layers of multi-scale architectures.
arXiv Detail & Related papers (2024-04-11T15:39:10Z) - Joint Channel Estimation and Feedback with Masked Token Transformers in
Massive MIMO Systems [74.52117784544758]
This paper proposes an encoder-decoder based network that unveils the intrinsic frequency-domain correlation within the CSI matrix.
The entire encoder-decoder network is utilized for channel compression.
Our method outperforms state-of-the-art channel estimation and feedback techniques in joint tasks.
arXiv Detail & Related papers (2023-06-08T06:15:17Z) - Optimal channel selection with discrete QCQP [14.734454356396158]
We propose a novel channel selection method that optimally selects channels via discrete QCQP.
We also propose a quadratic model that accurately estimates the actual inference time of the pruned network.
Our experiments on CIFAR-10 and ImageNet show our proposed pruning method outperforms other fixed-importance channel pruning methods on various network architectures.
arXiv Detail & Related papers (2022-02-24T23:26:51Z) - Group Fisher Pruning for Practical Network Compression [58.25776612812883]
We present a general channel pruning approach that can be applied to various complicated structures.
We derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels.
Our method can be used to prune any structures including those with coupled channels.
arXiv Detail & Related papers (2021-08-02T08:21:44Z) - AirNet: Neural Network Transmission over the Air [20.45405359815043]
State-of-the-art performance for many edge applications is achieved by deep neural networks (DNNs)
In this paper, we introduce AirNet, a family of novel training and transmission methods.
AirNet allows DNNs to be efficiently delivered over wireless channels under stringent transmit power and latency constraints.
arXiv Detail & Related papers (2021-05-24T09:16:04Z) - UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks
Compression and Acceleration [24.42067007684169]
We propose a novel uniform channel pruning (UCP) method to prune deep CNN.
The unimportant channels, including convolutional kernels related to them, are pruned directly.
We verify our method on CIFAR-10, CIFAR-100 and ILSVRC-2012 for image classification.
arXiv Detail & Related papers (2020-10-03T01:51:06Z) - Deep Denoising Neural Network Assisted Compressive Channel Estimation
for mmWave Intelligent Reflecting Surfaces [99.34306447202546]
This paper proposes a deep denoising neural network assisted compressive channel estimation for mmWave IRS systems.
We first introduce a hybrid passive/active IRS architecture, where very few receive chains are employed to estimate the uplink user-to-IRS channels.
The complete channel matrix can be reconstructed from the limited measurements based on compressive sensing.
arXiv Detail & Related papers (2020-06-03T12:18:57Z) - PruneNet: Channel Pruning via Global Importance [22.463154358632472]
We propose a simple-yet-effective method for pruning channels based on a computationally light-weight yet effective data driven optimization step.
With non-uniform pruning across the layers on ResNet-$50$, we are able to match the FLOP reduction of state-of-the-art channel pruning results.
arXiv Detail & Related papers (2020-05-22T17:09:56Z) - Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs.
Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2020-04-06T15:51:00Z) - Decentralized Learning for Channel Allocation in IoT Networks over
Unlicensed Bandwidth as a Contextual Multi-player Multi-armed Bandit Game [134.88020946767404]
We study a decentralized channel allocation problem in an ad-hoc Internet of Things network underlaying on the spectrum licensed to a primary cellular network.
Our study maps this problem into a contextual multi-player, multi-armed bandit game, and proposes a purely decentralized, three-stage policy learning algorithm through trial-and-error.
arXiv Detail & Related papers (2020-03-30T10:05:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.