Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing

• URL: http://arxiv.org/abs/2309.17211v1
• Date: Fri, 29 Sep 2023 13:09:40 GMT
• Title: Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing
• Authors: Lukas Meiner, Jens Mehnert, Alexandru Paul Condurache
• Abstract summary: We propose HASTE (Hashing for Tractable Efficiency), a parameter-free and data-free module that acts as a plug-and-play replacement for any regular convolution module. We are able to drastically compress latent feature maps without sacrificing much accuracy by using locality-sensitive hashing (LSH) In particular, we are able to instantly drop 46.72% of FLOPs while only losing 1.25% accuracy by just swapping the convolution modules in a ResNet34 on CIFAR-10 for our HASTE module.
• Score: 50.79602839359522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To reduce the computational cost of convolutional neural networks (CNNs) for usage on resource-constrained devices, structured pruning approaches have shown promising results, drastically reducing floating-point operations (FLOPs) without substantial drops in accuracy. However, most recent methods require fine-tuning or specific training procedures to achieve a reasonable trade-off between retained accuracy and reduction in FLOPs. This introduces additional cost in the form of computational overhead and requires training data to be available. To this end, we propose HASTE (Hashing for Tractable Efficiency), a parameter-free and data-free module that acts as a plug-and-play replacement for any regular convolution module. It instantly reduces the network's test-time inference cost without requiring any training or fine-tuning. We are able to drastically compress latent feature maps without sacrificing much accuracy by using locality-sensitive hashing (LSH) to detect redundancies in the channel dimension. Similar channels are aggregated to reduce the input and filter depth simultaneously, allowing for cheaper convolutions. We demonstrate our approach on the popular vision benchmarks CIFAR-10 and ImageNet. In particular, we are able to instantly drop 46.72% of FLOPs while only losing 1.25% accuracy by just swapping the convolution modules in a ResNet34 on CIFAR-10 for our HASTE module.

Related papers

• Joint Pruning and Channel-wise Mixed-Precision Quantization for Efficient Deep Neural Networks [10.229120811024162]
  deep neural networks (DNNs) pose significant challenges to their deployment on edge devices. Common approaches to address this issue are pruning and mixed-precision quantization. We propose a novel methodology to apply them jointly via a lightweight gradient-based search.
  arXiv  Detail & Related papers  (2024-07-01T08:07:02Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• Attention-based Feature Compression for CNN Inference Offloading in Edge Computing [93.67044879636093]
  This paper studies the computational offloading of CNN inference in device-edge co-inference systems. We propose a novel autoencoder-based CNN architecture (AECNN) for effective feature extraction at end-device. Experiments show that AECNN can compress the intermediate data by more than 256x with only about 4% accuracy loss.
  arXiv  Detail & Related papers  (2022-11-24T18:10:01Z)
• Design of High-Throughput Mixed-Precision CNN Accelerators on FPGA [0.0]
  Layer-wise mixed-precision quantization allows for more efficient results while inflating the design space. We present an in-depth quantitative methodology to efficiently explore the design space considering the limited hardware resources of a given FPGA. Our resulting hardware accelerators implement truly mixed-precision operations that enable efficient execution of layer-wise and channel-wise quantized CNNs.
  arXiv  Detail & Related papers  (2022-08-09T15:32:51Z)
• DIVISION: Memory Efficient Training via Dual Activation Precision [60.153754740511864]
  State-of-the-art work combines a search of quantization bit-width with the training, which makes the procedure complicated and less transparent. We propose a simple and effective method to compress DNN training. Experiment results show DIVISION has better comprehensive performance than state-of-the-art methods, including over 10x compression of activation maps and competitive training throughput, without loss of model accuracy.
  arXiv  Detail & Related papers  (2022-08-05T03:15:28Z)
• Neural Network Pruning by Cooperative Coevolution [16.0753044050118]
  We propose a new filter pruning algorithm CCEP by cooperative coevolution. CCEP reduces the pruning space by a divide-and-conquer strategy. Experiments show that CCEP can achieve a competitive performance with the state-of-the-art pruning methods.
  arXiv  Detail & Related papers  (2022-04-12T09:06:38Z)
• GDP: Stabilized Neural Network Pruning via Gates with Differentiable Polarization [84.57695474130273]
  Gate-based or importance-based pruning methods aim to remove channels whose importance is smallest. GDP can be plugged before convolutional layers without bells and whistles, to control the on-and-off of each channel. Experiments conducted over CIFAR-10 and ImageNet datasets show that the proposed GDP achieves the state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-09-06T03:17:10Z)
• Structured Directional Pruning via Perturbation Orthogonal Projection [13.704348351073147]
  A more reasonable approach is to find a sparse minimizer along the flat minimum valley found byNIST. We propose the structured directional pruning based on projecting the perturbations onto the flat minimum valley. Experiments show that our method obtains the state-of-the-art pruned accuracy (i.e. 93.97% on VGG16, CIFAR-10 task) without retraining.
  arXiv  Detail & Related papers  (2021-07-12T11:35:47Z)
• Toward Compact Deep Neural Networks via Energy-Aware Pruning [2.578242050187029]
  We propose a novel energy-aware pruning method that quantifies the importance of each filter in the network using nuclear-norm (NN) We achieve competitive results with 40.4/49.8% of FLOPs and 45.9/52.9% of parameter reduction with 94.13/94.61% in the Top-1 accuracy with ResNet-56/110 on CIFAR-10.
  arXiv  Detail & Related papers  (2021-03-19T15:33:16Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)
• ACP: Automatic Channel Pruning via Clustering and Swarm Intelligence Optimization for CNN [6.662639002101124]
  convolutional neural network (CNN) gets deeper and wider in recent years. Existing magnitude-based pruning methods are efficient, but the performance of the compressed network is unpredictable. We propose a novel automatic channel pruning method (ACP) ACP is evaluated against several state-of-the-art CNNs on three different classification datasets.
  arXiv  Detail & Related papers  (2021-01-16T08:56:38Z)
• UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks Compression and Acceleration [24.42067007684169]
  We propose a novel uniform channel pruning (UCP) method to prune deep CNN. The unimportant channels, including convolutional kernels related to them, are pruned directly. We verify our method on CIFAR-10, CIFAR-100 and ILSVRC-2012 for image classification.
  arXiv  Detail & Related papers  (2020-10-03T01:51:06Z)
• AQD: Towards Accurate Fully-Quantized Object Detection [94.06347866374927]
  We propose an Accurate Quantized object Detection solution, termed AQD, to get rid of floating-point computation. Our AQD achieves comparable or even better performance compared with the full-precision counterpart under extremely low-bit schemes.
  arXiv  Detail & Related papers  (2020-07-14T09:07:29Z)
• Enabling On-Device CNN Training by Self-Supervised Instance Filtering and Error Map Pruning [17.272561332310303]
  This work aims to enable on-device training of convolutional neural networks (CNNs) by reducing the computation cost at training time. CNN models are usually trained on high-performance computers and only the trained models are deployed to edge devices.
  arXiv  Detail & Related papers  (2020-07-07T05:52:37Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks [6.534515590778012]
  Pruning is one of the predominant approaches used for deep network compression. We present a simple-yet-effective gradual channel pruning while training methodology using a novel data-driven metric. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet.
  arXiv  Detail & Related papers  (2020-02-23T17:56:18Z)
• Filter Sketch for Network Pruning [184.41079868885265]
  We propose a novel network pruning approach by information preserving of pre-trained network weights (filters) Our approach, referred to as FilterSketch, encodes the second-order information of pre-trained weights. Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of FLOPs and prunes 59.9% of network parameters with negligible accuracy cost.
  arXiv  Detail & Related papers  (2020-01-23T13:57:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.