BiBench: Benchmarking and Analyzing Network Binarization
- URL: http://arxiv.org/abs/2301.11233v2
- Date: Sat, 20 May 2023 11:04:19 GMT
- Title: BiBench: Benchmarking and Analyzing Network Binarization
- Authors: Haotong Qin, Mingyuan Zhang, Yifu Ding, Aoyu Li, Zhongang Cai, Ziwei
Liu, Fisher Yu, Xianglong Liu
- Abstract summary: Network binarization emerges as one of the most promising compression approaches offering extraordinary computation and memory savings.
Common challenges of binarization, such as accuracy degradation and efficiency limitation, suggest that its attributes are not fully understood.
We present BiBench, a rigorously designed benchmark with in-depth analysis for network binarization.
- Score: 72.59760752906757
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Network binarization emerges as one of the most promising compression
approaches offering extraordinary computation and memory savings by minimizing
the bit-width. However, recent research has shown that applying existing
binarization algorithms to diverse tasks, architectures, and hardware in
realistic scenarios is still not straightforward. Common challenges of
binarization, such as accuracy degradation and efficiency limitation, suggest
that its attributes are not fully understood. To close this gap, we present
BiBench, a rigorously designed benchmark with in-depth analysis for network
binarization. We first carefully scrutinize the requirements of binarization in
the actual production and define evaluation tracks and metrics for a
comprehensive and fair investigation. Then, we evaluate and analyze a series of
milestone binarization algorithms that function at the operator level and with
extensive influence. Our benchmark reveals that 1) the binarized operator has a
crucial impact on the performance and deployability of binarized networks; 2)
the accuracy of binarization varies significantly across different learning
tasks and neural architectures; 3) binarization has demonstrated promising
efficiency potential on edge devices despite the limited hardware support. The
results and analysis also lead to a promising paradigm for accurate and
efficient binarization. We believe that BiBench will contribute to the broader
adoption of binarization and serve as a foundation for future research. The
code for our BiBench is released https://github.com/htqin/BiBench .
Related papers
- BiFSMNv2: Pushing Binary Neural Networks for Keyword Spotting to
Real-Network Performance [54.214426436283134]
Deep neural networks, such as the Deep-FSMN, have been widely studied for keyword spotting (KWS) applications.
We present a strong yet efficient binary neural network for KWS, namely BiFSMNv2, pushing it to the real-network accuracy performance.
We highlight that benefiting from the compact architecture and optimized hardware kernel, BiFSMNv2 can achieve an impressive 25.1x speedup and 20.2x storage-saving on edge hardware.
arXiv Detail & Related papers (2022-11-13T18:31:45Z) - Network Binarization via Contrastive Learning [16.274341164897827]
We establish a novel contrastive learning framework while training Binary Neural Networks (BNNs)
MI is introduced as the metric to measure the information shared between binary and FP activations.
Results show that our method can be implemented as a pile-up module on existing state-of-the-art binarization methods.
arXiv Detail & Related papers (2022-07-06T21:04:53Z) - BiFSMN: Binary Neural Network for Keyword Spotting [47.46397208920726]
BiFSMN is an accurate and extreme-efficient binary neural network for KWS.
We show that BiFSMN can achieve an impressive 22.3x speedup and 15.5x storage-saving on real-world edge hardware.
arXiv Detail & Related papers (2022-02-14T05:16:53Z) - Semantic-aware Binary Code Representation with BERT [27.908093567605484]
A wide range of binary analysis applications, such as bug discovery, malware analysis and code clone detection, require recovery of contextual meanings on a binary code.
Recently, binary analysis techniques based on machine learning have been proposed to automatically reconstruct the code representation of a binary.
In this paper, we propose DeepSemantic utilizing BERT in producing the semantic-aware code representation of a binary code.
arXiv Detail & Related papers (2021-06-10T03:31:29Z) - High-Capacity Expert Binary Networks [56.87581500474093]
Network binarization is a promising hardware-aware direction for creating efficient deep models.
Despite its memory and computational advantages, reducing the accuracy gap between binary models and their real-valued counterparts remains an unsolved challenging research problem.
We propose Expert Binary Convolution, which, for the first time, tailors conditional computing to binary networks by learning to select one data-specific expert binary filter at a time conditioned on input features.
arXiv Detail & Related papers (2020-10-07T17:58:10Z) - Binary Neural Networks: A Survey [126.67799882857656]
The binary neural network serves as a promising technique for deploying deep models on resource-limited devices.
The binarization inevitably causes severe information loss, and even worse, its discontinuity brings difficulty to the optimization of the deep network.
We present a survey of these algorithms, mainly categorized into the native solutions directly conducting binarization, and the optimized ones using techniques like minimizing the quantization error, improving the network loss function, and reducing the gradient error.
arXiv Detail & Related papers (2020-03-31T16:47:20Z) - ReActNet: Towards Precise Binary Neural Network with Generalized
Activation Functions [76.05981545084738]
We propose several ideas for enhancing a binary network to close its accuracy gap from real-valued networks without incurring any additional computational cost.
We first construct a baseline network by modifying and binarizing a compact real-valued network with parameter-free shortcuts.
We show that the proposed ReActNet outperforms all the state-of-the-arts by a large margin.
arXiv Detail & Related papers (2020-03-07T02:12:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.