Self-Distribution Binary Neural Networks

• URL: http://arxiv.org/abs/2103.02394v2
• Date: Thu, 4 Mar 2021 02:17:03 GMT
• Title: Self-Distribution Binary Neural Networks
• Authors: Ping Xue, Yang Lu, Jingfei Chang, Xing Wei, Zhen Wei
• Abstract summary: We study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation) We propose Self-Distribution Binary Neural Network (SD-BNN) Experiments on CIFAR-10 and ImageNet datasets show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs.
• Score: 18.69165083747967
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation). Feature representation is critical for deep neural networks, while in BNNs, the features only differ in signs. Prior work introduces scaling factors into binary weights and activations to reduce the quantization error and effectively improves the classification accuracy of BNNs. However, the scaling factors not only increase the computational complexity of networks, but also make no sense to the signs of binary features. To this end, Self-Distribution Binary Neural Network (SD-BNN) is proposed. Firstly, we utilize Activation Self Distribution (ASD) to adaptively adjust the sign distribution of activations, thereby improve the sign differences of the outputs of the convolution. Secondly, we adjust the sign distribution of weights through Weight Self Distribution (WSD) and then fine-tune the sign distribution of the outputs of the convolution. Extensive experiments on CIFAR-10 and ImageNet datasets with various network structures show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs (e.g., achieves 92.5% on CIFAR-10 and 66.5% on ImageNet with ResNet-18) with less computation cost. Code is available at https://github.com/ pingxue-hfut/SD-BNN.

Related papers

• Boosting Binary Neural Networks via Dynamic Thresholds Learning [21.835748440099586]
  We introduce DySign to reduce information loss and boost representative capacity of BNNs. For DCNNs, DyBCNNs based on two backbones achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset. For ViTs, DyCCT presents the superiority of the convolutional embedding layer in fully binarized ViTs and 56.1% on the ImageNet dataset.
  arXiv  Detail & Related papers  (2022-11-04T07:18:21Z)
• Basic Binary Convolution Unit for Binarized Image Restoration Network [146.0988597062618]
  In this study, we reconsider components in binary convolution, such as residual connection, BatchNorm, activation function, and structure, for image restoration tasks. Based on our findings and analyses, we design a simple yet efficient basic binary convolution unit (BBCU) Our BBCU significantly outperforms other BNNs and lightweight models, which shows that BBCU can serve as a basic unit for binarized IR networks.
  arXiv  Detail & Related papers  (2022-10-02T01:54:40Z)
• Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies [52.691032025163175]
  Existing Binary Neural Networks (BNNs) operate mainly on local convolutions with binarization function. We present new designs of binary neural modules, which enables leading binary neural modules by a large margin.
  arXiv  Detail & Related papers  (2022-09-03T11:51:04Z)
• Elastic-Link for Binarized Neural Network [9.83865304744923]
  "Elastic-Link" (EL) module enrich information flow within a BNN by adaptively adding real-valued input features to the subsequent convolutional output features. EL produces a significant improvement on the challenging large-scale ImageNet dataset. With the integration of ReActNet, it yields a new state-of-the-art result of 71.9% top-1 accuracy.
  arXiv  Detail & Related papers  (2021-12-19T13:49:29Z)
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks [72.81092567651395]
  Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs. SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space. Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
  arXiv  Detail & Related papers  (2021-10-18T11:30:29Z)
• Dynamic Binary Neural Network by learning channel-wise thresholds [9.432747511001246]
  We propose a dynamic BNN (DyBNN) incorporating dynamic learnable channel-wise thresholds of Sign function and shift parameters of PReLU. The DyBNN based on two backbones of ReActNet (MobileNetV1 and ResNet18) achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset.
  arXiv  Detail & Related papers  (2021-10-08T17:41:36Z)
• "BNN - BN = ?": Training Binary Neural Networks without Batch Normalization [92.23297927690149]
  Batch normalization (BN) is a key facilitator and considered essential for state-of-the-art binary neural networks (BNN) We extend their framework to training BNNs, and for the first time demonstrate that BNs can be completed removed from BNN training and inference regimes.
  arXiv  Detail & Related papers  (2021-04-16T16:46:57Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• FracBNN: Accurate and FPGA-Efficient Binary Neural Networks with Fractional Activations [20.218382369944152]
  Binary neural networks (BNNs) have 1-bit weights and activations. BNNs tend to produce a much lower accuracy on realistic datasets such as ImageNet. This work proposes FracBNN, which exploits fractional activations to substantially improve the accuracy of BNNs.
  arXiv  Detail & Related papers  (2020-12-22T17:49:30Z)
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution [12.46127622357824]
  We show that unbalanced activation distribution can actually improve the accuracy of BNNs. We also show that adjusting the threshold values of binary activation functions results in the unbalanced distribution of the binary activation. Experimental results show that the accuracy of previous BNN models can be improved by simply shifting the threshold values of binary activation functions.
  arXiv  Detail & Related papers  (2020-12-02T02:49:53Z)
• Distillation Guided Residual Learning for Binary Convolutional Neural Networks [83.6169936912264]
  It is challenging to bridge the performance gap between Binary CNN (BCNN) and Floating point CNN (FCNN) We observe that, this performance gap leads to substantial residuals between intermediate feature maps of BCNN and FCNN. To minimize the performance gap, we enforce BCNN to produce similar intermediate feature maps with the ones of FCNN. This training strategy, i.e., optimizing each binary convolutional block with block-wise distillation loss derived from FCNN, leads to a more effective optimization to BCNN.
  arXiv  Detail & Related papers  (2020-07-10T07:55:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.