OvSW: Overcoming Silent Weights for Accurate Binary Neural Networks

• URL: http://arxiv.org/abs/2407.05257v1
• Date: Sun, 7 Jul 2024 05:01:20 GMT
• Title: OvSW: Overcoming Silent Weights for Accurate Binary Neural Networks
• Authors: Jingyang Xiang, Zuohui Chen, Siqi Li, Qing Wu, Yong Liu,
• Abstract summary: We investigate the efficiency of weight sign updates in Binary Neural Networks(BNNs) For vanilla BNNs, over 50% of the weights remain their signs unchanged during training. We propose Overcome Silent Weights(OvSW) to address the issue.
• Score: 19.41917323210239
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Binary Neural Networks~(BNNs) have been proven to be highly effective for deploying deep neural networks on mobile and embedded platforms. Most existing works focus on minimizing quantization errors, improving representation ability, or designing gradient approximations to alleviate gradient mismatch in BNNs, while leaving the weight sign flipping, a critical factor for achieving powerful BNNs, untouched. In this paper, we investigate the efficiency of weight sign updates in BNNs. We observe that, for vanilla BNNs, over 50\% of the weights remain their signs unchanged during training, and these weights are not only distributed at the tails of the weight distribution but also universally present in the vicinity of zero. We refer to these weights as ``silent weights'', which slow down convergence and lead to a significant accuracy degradation. Theoretically, we reveal this is due to the independence of the BNNs gradient from the latent weight distribution. To address the issue, we propose Overcome Silent Weights~(OvSW). OvSW first employs Adaptive Gradient Scaling~(AGS) to establish a relationship between the gradient and the latent weight distribution, thereby improving the overall efficiency of weight sign updates. Additionally, we design Silence Awareness Decaying~(SAD) to automatically identify ``silent weights'' by tracking weight flipping state, and apply an additional penalty to ``silent weights'' to facilitate their flipping. By efficiently updating weight signs, our method achieves faster convergence and state-of-the-art performance on CIFAR10 and ImageNet1K dataset with various architectures. For example, OvSW obtains 61.6\% and 65.5\% top-1 accuracy on the ImageNet1K using binarized ResNet18 and ResNet34 architecture respectively. Codes are available at \url{https://github.com/JingyangXiang/OvSW}.

Related papers

• Improved Generalization of Weight Space Networks via Augmentations [56.571475005291035]
  Learning in deep weight spaces (DWS) is an emerging research direction, with applications to 2D and 3D neural fields (INRs, NeRFs) We empirically analyze the reasons for this overfitting and find that a key reason is the lack of diversity in DWS datasets. To address this, we explore strategies for data augmentation in weight spaces and propose a MixUp method adapted for weight spaces.
  arXiv  Detail & Related papers  (2024-02-06T15:34:44Z)
• BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation [116.26521375592759]
  Quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation. Extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration.
  arXiv  Detail & Related papers  (2022-07-04T13:25:49Z)
• Low-Precision Training in Logarithmic Number System using Multiplicative Weight Update [49.948082497688404]
  Training large-scale deep neural networks (DNNs) currently requires a significant amount of energy, leading to serious environmental impacts. One promising approach to reduce the energy costs is representing DNNs with low-precision numbers. We jointly design a lowprecision training framework involving a logarithmic number system (LNS) and a multiplicative weight update training method, termed LNS-Madam.
  arXiv  Detail & Related papers  (2021-06-26T00:32:17Z)
• ReCU: Reviving the Dead Weights in Binary Neural Networks [153.6789340484509]
  We explore the influence of "dead weights" which refer to a group of weights that are barely updated during the training of BNNs. We prove that reviving the "dead weights" by ReCU can result in a smaller quantization error. Our method offers not only faster BNN training, but also state-of-the-art performance on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2021-03-23T08:11:20Z)
• Self-Distribution Binary Neural Networks [18.69165083747967]
  We study the binary neural networks (BNNs) of which both the weights and activations are binary (i.e., 1-bit representation) We propose Self-Distribution Binary Neural Network (SD-BNN) Experiments on CIFAR-10 and ImageNet datasets show that the proposed SD-BNN consistently outperforms the state-of-the-art (SOTA) BNNs.
  arXiv  Detail & Related papers  (2021-03-03T13:39:52Z)
• SiMaN: Sign-to-Magnitude Network Binarization [165.5630656849309]
  We show that our weight binarization provides an analytical solution by encoding high-magnitude weights into +1s, and 0s otherwise. We prove that the learned weights of binarized networks roughly follow a Laplacian distribution that does not allow entropy. Our method, dubbed sign-to- neural network binarization (SiMaN), is evaluated on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2021-02-16T07:03:51Z)
• The Implicit Biases of Stochastic Gradient Descent on Deep Neural Networks with Batch Normalization [44.30960913470372]
  Deep neural networks with batch normalization (BN-DNNs) are invariant to weight rescaling due to their normalization operations. We investigate the implicit biases of gradient descent (SGD) on BN-DNNs to provide a theoretical explanation for the efficacy of weight decay.
  arXiv  Detail & Related papers  (2021-02-06T03:40:20Z)
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks [73.29587731448345]
  This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. Second, to obtain low bit-width activations, existing works consider all channels equally.
  arXiv  Detail & Related papers  (2020-12-26T15:21:18Z)
• Sparsity-Control Ternary Weight Networks [34.00378876525579]
  We focus on training ternary weight -1, 0, +1 networks which can avoid multiplications and dramatically reduce the memory and requirements. Existing approaches to training ternary weight networks cannot control the sparsity of the ternary weights. We propose the first sparsity-control approach (SCA) to training ternary weight networks.
  arXiv  Detail & Related papers  (2020-11-01T18:06:26Z)
• Train-by-Reconnect: Decoupling Locations of Weights from their Values [6.09170287691728]
  We show that untrained deep neural networks (DNNs) are different from trained ones. We propose a novel method named Lookahead Permutation (LaPerm) to train DNNs by reconnecting the weights. When the initial weights share a single value, our method finds weight neural network with far better-than-chance accuracy.
  arXiv  Detail & Related papers  (2020-03-05T12:40:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.