ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation

• URL: http://arxiv.org/abs/2407.02881v1
• Date: Wed, 3 Jul 2024 07:56:51 GMT
• Title: ShiftAddAug: Augment Multiplication-Free Tiny Neural Network with Hybrid Computation
• Authors: Yipin Guo, Zihao Li, Yilin Lang, Qinyuan Ren,
• Abstract summary: ShiftAddAug uses costly multiplication to augment efficient but less powerful multiplication-free operators, improving performance without any inference overhead. It secures up to a 4.95% increase in accuracy on the CIFAR100 compared to its directly trained counterparts, even surpassing the performance of multiplicative NNs.
• Score: 7.143509813507735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Operators devoid of multiplication, such as Shift and Add, have gained prominence for their compatibility with hardware. However, neural networks (NNs) employing these operators typically exhibit lower accuracy compared to conventional NNs with identical structures. ShiftAddAug uses costly multiplication to augment efficient but less powerful multiplication-free operators, improving performance without any inference overhead. It puts a ShiftAdd tiny NN into a large multiplicative model and encourages it to be trained as a sub-model to obtain additional supervision. In order to solve the weight discrepancy problem between hybrid operators, a new weight sharing method is proposed. Additionally, a novel two stage neural architecture search is used to obtain better augmentation effects for smaller but stronger multiplication-free tiny neural networks. The superiority of ShiftAddAug is validated through experiments in image classification and semantic segmentation, consistently delivering noteworthy enhancements. Remarkably, it secures up to a 4.95% increase in accuracy on the CIFAR100 compared to its directly trained counterparts, even surpassing the performance of multiplicative NNs.

Related papers

• DenseShift: Towards Accurate and Efficient Low-Bit Power-of-Two Quantization [27.231327287238102]
  We propose the DenseShift network, which significantly improves the accuracy of Shift networks. Our experiments on various computer vision and speech tasks demonstrate that DenseShift outperforms existing low-bit multiplication-free networks.
  arXiv  Detail & Related papers  (2022-08-20T15:17:40Z)
• ShiftAddNAS: Hardware-Inspired Search for More Accurate and Efficient Neural Networks [42.28659737268829]
  ShiftAddNAS can automatically search for more accurate and more efficient NNs. ShiftAddNAS integrates the first hybrid search space that incorporates both multiplication-based and multiplication-free operators. Experiments and ablation studies consistently validate the efficacy of ShiftAddNAS.
  arXiv  Detail & Related papers  (2022-05-17T06:40:13Z)
• An Empirical Study of Adder Neural Networks for Object Detection [67.64041181937624]
  Adder neural networks (AdderNets) have shown impressive performance on image classification with only addition operations. We present an empirical study of AdderNets for object detection.
  arXiv  Detail & Related papers  (2021-12-27T11:03:13Z)
• Adder Neural Networks [75.54239599016535]
  We present adder networks (AdderNets) to trade massive multiplications in deep neural networks. In AdderNets, we take the $ell_p$-norm distance between filters and input feature as the output response. We show that the proposed AdderNets can achieve 75.7% Top-1 accuracy 92.3% Top-5 accuracy using ResNet-50 on the ImageNet dataset.
  arXiv  Detail & Related papers  (2021-05-29T04:02:51Z)
• Winograd Algorithm for AdderNet [54.93995545896655]
  Adder neural network (AdderNet) is a new kind of deep model that replaces the original massive multiplications in convolutions by additions. This paper studies the winograd algorithm, which is a widely used fast algorithm for accelerating convolution and saving the computational costs.
  arXiv  Detail & Related papers  (2021-05-12T09:13:34Z)
• ShiftAddNet: A Hardware-Inspired Deep Network [87.18216601210763]
  ShiftAddNet is an energy-efficient multiplication-less deep neural network. It leads to both energy-efficient inference and training, without compromising expressive capacity. ShiftAddNet aggressively reduces over 80% hardware-quantified energy cost of DNNs training and inference, while offering comparable or better accuracies.
  arXiv  Detail & Related papers  (2020-10-24T05:09:14Z)
• AdderNet: Do We Really Need Multiplications in Deep Learning? [159.174891462064]
  We present adder networks (AdderNets) to trade massive multiplications in deep neural networks for much cheaper additions to reduce computation costs. We develop a special back-propagation approach for AdderNets by investigating the full-precision gradient. As a result, the proposed AdderNets can achieve 74.9% Top-1 accuracy 91.7% Top-5 accuracy using ResNet-50 on the ImageNet dataset.
  arXiv  Detail & Related papers  (2019-12-31T06:56:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.