Boosting Binary Neural Networks via Dynamic Thresholds Learning
- URL: http://arxiv.org/abs/2211.02292v1
- Date: Fri, 4 Nov 2022 07:18:21 GMT
- Title: Boosting Binary Neural Networks via Dynamic Thresholds Learning
- Authors: Jiehua Zhang, Xueyang Zhang, Zhuo Su, Zitong Yu, Yanghe Feng, Xin Lu,
Matti Pietik\"ainen, Li Liu
- Abstract summary: We introduce DySign to reduce information loss and boost representative capacity of BNNs.
For DCNNs, DyBCNNs based on two backbones achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset.
For ViTs, DyCCT presents the superiority of the convolutional embedding layer in fully binarized ViTs and 56.1% on the ImageNet dataset.
- Score: 21.835748440099586
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Developing lightweight Deep Convolutional Neural Networks (DCNNs) and Vision
Transformers (ViTs) has become one of the focuses in vision research since the
low computational cost is essential for deploying vision models on edge
devices. Recently, researchers have explored highly computational efficient
Binary Neural Networks (BNNs) by binarizing weights and activations of
Full-precision Neural Networks. However, the binarization process leads to an
enormous accuracy gap between BNN and its full-precision version. One of the
primary reasons is that the Sign function with predefined or learned static
thresholds limits the representation capacity of binarized architectures since
single-threshold binarization fails to utilize activation distributions. To
overcome this issue, we introduce the statistics of channel information into
explicit thresholds learning for the Sign Function dubbed DySign to generate
various thresholds based on input distribution. Our DySign is a straightforward
method to reduce information loss and boost the representative capacity of
BNNs, which can be flexibly applied to both DCNNs and ViTs (i.e., DyBCNN and
DyBinaryCCT) to achieve promising performance improvement. As shown in our
extensive experiments. For DCNNs, DyBCNNs based on two backbones (MobileNetV1
and ResNet18) achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset,
outperforming baselines by a large margin (i.e., 1.8% and 1.5% respectively).
For ViTs, DyBinaryCCT presents the superiority of the convolutional embedding
layer in fully binarized ViTs and achieves 56.1% on the ImageNet dataset, which
is nearly 9% higher than the baseline.
Related papers
- ReActXGB: A Hybrid Binary Convolutional Neural Network Architecture for Improved Performance and Computational Efficiency [0.0]
We propose a hybrid model named ReActXGB, where we replace the fully convolutional layer of ReActNet-A with XGBoost.
This modification targets to narrow the performance gap between BCNNs and real-valued networks while maintaining lower computational costs.
arXiv Detail & Related papers (2024-05-11T16:38:50Z) - Recurrent Bilinear Optimization for Binary Neural Networks [58.972212365275595]
BNNs neglect the intrinsic bilinear relationship of real-valued weights and scale factors.
Our work is the first attempt to optimize BNNs from the bilinear perspective.
We obtain robust RBONNs, which show impressive performance over state-of-the-art BNNs on various models and datasets.
arXiv Detail & Related papers (2022-09-04T06:45:33Z) - INSTA-BNN: Binary Neural Network with INSTAnce-aware Threshold [16.890849856271185]
We propose a novel BNN design called Binary Neural Network with INSTAnce-aware threshold (INSTA-BNN)
INSTA-BNN controls the quantization threshold dynamically in an input-dependent or instance-aware manner.
Our study shows that INSTA-BNN outperforms the baseline by 3.0% and 2.8% on the ImageNet classification task with comparable computing cost.
arXiv Detail & Related papers (2022-04-15T12:30:02Z) - Elastic-Link for Binarized Neural Network [9.83865304744923]
"Elastic-Link" (EL) module enrich information flow within a BNN by adaptively adding real-valued input features to the subsequent convolutional output features.
EL produces a significant improvement on the challenging large-scale ImageNet dataset.
With the integration of ReActNet, it yields a new state-of-the-art result of 71.9% top-1 accuracy.
arXiv Detail & Related papers (2021-12-19T13:49:29Z) - Sub-bit Neural Networks: Learning to Compress and Accelerate Binary
Neural Networks [72.81092567651395]
Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs.
SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space.
Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
arXiv Detail & Related papers (2021-10-18T11:30:29Z) - Dynamic Binary Neural Network by learning channel-wise thresholds [9.432747511001246]
We propose a dynamic BNN (DyBNN) incorporating dynamic learnable channel-wise thresholds of Sign function and shift parameters of PReLU.
The DyBNN based on two backbones of ReActNet (MobileNetV1 and ResNet18) achieve 71.2% and 67.4% top1-accuracy on ImageNet dataset.
arXiv Detail & Related papers (2021-10-08T17:41:36Z) - Distribution-sensitive Information Retention for Accurate Binary Neural
Network [49.971345958676196]
We present a novel Distribution-sensitive Information Retention Network (DIR-Net) to retain the information of the forward activations and backward gradients.
Our DIR-Net consistently outperforms the SOTA binarization approaches under mainstream and compact architectures.
We conduct our DIR-Net on real-world resource-limited devices which achieves 11.1 times storage saving and 5.4 times speedup.
arXiv Detail & Related papers (2021-09-25T10:59:39Z) - Self-Distribution Binary Neural Networks [18.69165083747967]
We study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation)
We propose Self-Distribution Binary Neural Network (SD-BNN)
Experiments on CIFAR-10 and ImageNet datasets show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs.
arXiv Detail & Related papers (2021-03-03T13:39:52Z) - S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural
Networks via Guided Distribution Calibration [74.5509794733707]
We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution.
Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs.
Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
arXiv Detail & Related papers (2021-02-17T18:59:28Z) - ReActNet: Towards Precise Binary Neural Network with Generalized
Activation Functions [76.05981545084738]
We propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost.
We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts.
We show that the proposed ReActNet outperforms all the state-of-the-arts by a large margin.
arXiv Detail & Related papers (2020-03-07T02:12:02Z) - Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters.
Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques.
We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
arXiv Detail & Related papers (2020-02-03T04:11:13Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.