Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing

• URL: http://arxiv.org/abs/2011.03891v2
• Date: Mon, 21 Jun 2021 12:48:48 GMT
• Title: Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing
• Authors: Mengran Liu and Weiwei Fang and Xiaodong Ma and Wenyuan Xu and Naixue Xiong and Yi Ding
• Abstract summary: A critical challenge is to determine which channels are to be removed, so that the model accuracy will not be negatively affected. We propose a new attention module combining both spatial and channel attention. With the guidance of SCA, our CPSCA approach achieves higher inference accuracy than other state-of-the-art pruning methods.
• Score: 15.248962858090431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep Neural Networks (DNNs) have achieved remarkable success in many computer vision tasks recently, but the huge number of parameters and the high computation overhead hinder their deployments on resource-constrained edge devices. It is worth noting that channel pruning is an effective approach for compressing DNN models. A critical challenge is to determine which channels are to be removed, so that the model accuracy will not be negatively affected. In this paper, we first propose Spatial and Channel Attention (SCA), a new attention module combining both spatial and channel attention that respectively focuses on "where" and "what" are the most informative parts. Guided by the scale values generated by SCA for measuring channel importance, we further propose a new channel pruning approach called Channel Pruning guided by Spatial and Channel Attention (CPSCA). Experimental results indicate that SCA achieves the best inference accuracy, while incurring negligibly extra resource consumption, compared to other state-of-the-art attention modules. Our evaluation on two benchmark datasets shows that, with the guidance of SCA, our CPSCA approach achieves higher inference accuracy than other state-of-the-art pruning methods under the same pruning ratios.

Related papers

• Joint Channel Estimation and Feedback with Masked Token Transformers in Massive MIMO Systems [74.52117784544758]
  This paper proposes an encoder-decoder based network that unveils the intrinsic frequency-domain correlation within the CSI matrix. The entire encoder-decoder network is utilized for channel compression. Our method outperforms state-of-the-art channel estimation and feedback techniques in joint tasks.
  arXiv  Detail & Related papers  (2023-06-08T06:15:17Z)
• Visual Saliency-Guided Channel Pruning for Deep Visual Detectors in Autonomous Driving [3.236217153362305]
  Deep neural network (DNN) pruning has become a de facto component for deploying on resource-constrained devices. We propose a novel gradient-based saliency measure for visual detection and use it to guide our channel pruning. Experiments on the KITTI and COCO traffic datasets demonstrate our pruning method's efficacy and superiority over state-of-the-art competing approaches.
  arXiv  Detail & Related papers  (2023-03-04T22:08:22Z)
• BA-Net: Bridge Attention for Deep Convolutional Neural Networks [4.459779019776622]
  Bridge Attention Net (BA-Net) is proposed for better channel attention mechanisms. BA-Net can not only provide richer features to calculate channel weight when feedforward, but also multiply paths of parameters updating when backforward.
  arXiv  Detail & Related papers  (2021-12-08T07:39:18Z)
• CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
  We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference. We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms. Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
  arXiv  Detail & Related papers  (2021-10-21T06:26:31Z)
• Learning to Perform Downlink Channel Estimation in Massive MIMO Systems [72.76968022465469]
  We study downlink (DL) channel estimation in a Massive multiple-input multiple-output (MIMO) system. A common approach is to use the mean value as the estimate, motivated by channel hardening. We propose two novel estimation methods.
  arXiv  Detail & Related papers  (2021-09-06T13:42:32Z)
• Learning to Estimate RIS-Aided mmWave Channels [50.15279409856091]
  We focus on uplink cascaded channel estimation, where known and fixed base station combining and RIS phase control matrices are considered for collecting observations. To boost the estimation performance and reduce the training overhead, the inherent channel sparsity of mmWave channels is leveraged in the deep unfolding method. It is verified that the proposed deep unfolding network architecture can outperform the least squares (LS) method with a relatively smaller training overhead and online computational complexity.
  arXiv  Detail & Related papers  (2021-07-27T06:57:56Z)
• Channel-Based Attention for LCC Using Sentinel-2 Time Series [0.0]
  This paper proposes an architecture expressing predictions with respect to input channels. It relies on convolutional layers and an attention mechanism weighting the importance of each channel in the final classification decision. Experiments based on a Sentinel-2 SITS show promising results.
  arXiv  Detail & Related papers  (2021-03-31T06:24:15Z)
• Channelized Axial Attention for Semantic Segmentation [70.14921019774793]
  We propose the Channelized Axial Attention (CAA) to seamlessly integratechannel attention and axial attention with reduced computationalcomplexity. Our CAA not onlyrequires much less computation resources compared with otherdual attention models such as DANet, but also outperforms the state-of-the-art ResNet-101-based segmentation models on alltested datasets.
  arXiv  Detail & Related papers  (2021-01-19T03:08:03Z)
• Operation-Aware Soft Channel Pruning using Differentiable Masks [51.04085547997066]
  We propose a data-driven algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks.
  arXiv  Detail & Related papers  (2020-07-08T07:44:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.