BiPer: Binary Neural Networks using a Periodic Function

• URL: http://arxiv.org/abs/2404.01278v1
• Date: Mon, 1 Apr 2024 17:52:17 GMT
• Title: BiPer: Binary Neural Networks using a Periodic Function
• Authors: Edwin Vargas, Claudia Correa, Carlos Hinojosa, Henry Arguello,
• Abstract summary: Quantized neural networks employ reduced precision representations for both weights and activations. Binary Neural Networks (BNNs) are the extreme quantization case, representing values with just one bit. In contrast to current BNN approaches, we propose to employ a binary periodic (BiPer) function during binarization.
• Score: 17.461853355858022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantized neural networks employ reduced precision representations for both weights and activations. This quantization process significantly reduces the memory requirements and computational complexity of the network. Binary Neural Networks (BNNs) are the extreme quantization case, representing values with just one bit. Since the sign function is typically used to map real values to binary values, smooth approximations are introduced to mimic the gradients during error backpropagation. Thus, the mismatch between the forward and backward models corrupts the direction of the gradient, causing training inconsistency problems and performance degradation. In contrast to current BNN approaches, we propose to employ a binary periodic (BiPer) function during binarization. Specifically, we use a square wave for the forward pass to obtain the binary values and employ the trigonometric sine function with the same period of the square wave as a differentiable surrogate during the backward pass. We demonstrate that this approach can control the quantization error by using the frequency of the periodic function and improves network performance. Extensive experiments validate the effectiveness of BiPer in benchmark datasets and network architectures, with improvements of up to 1% and 0.69% with respect to state-of-the-art methods in the classification task over CIFAR-10 and ImageNet, respectively. Our code is publicly available at https://github.com/edmav4/BiPer.

Related papers

• Projected Stochastic Gradient Descent with Quantum Annealed Binary Gradients [51.82488018573326]
  We present QP-SBGD, a novel layer-wise optimiser tailored towards training neural networks with binary weights. BNNs reduce the computational requirements and energy consumption of deep learning models with minimal loss in accuracy. Our algorithm is implemented layer-wise, making it suitable to train larger networks on resource-limited quantum hardware.
  arXiv  Detail & Related papers  (2023-10-23T17:32:38Z)
• AdaBin: Improving Binary Neural Networks with Adaptive Binary Sets [27.022212653067367]
  This paper studies the Binary Neural Networks (BNNs) in which weights and activations are both binarized into 1-bit values. We present a simple yet effective approach called AdaBin to adaptively obtain the optimal binary sets. Experimental results on benchmark models and datasets demonstrate that the proposed AdaBin is able to achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-08-17T05:43:33Z)
• Network Binarization via Contrastive Learning [16.274341164897827]
  We establish a novel contrastive learning framework while training Binary Neural Networks (BNNs) MI is introduced as the metric to measure the information shared between binary and FP activations. Results show that our method can be implemented as a pile-up module on existing state-of-the-art binarization methods.
  arXiv  Detail & Related papers  (2022-07-06T21:04:53Z)
• Binarized Weight Error Networks With a Transition Regularization Term [4.56877715768796]
  This paper proposes a novel binarized weight network (BT) for a resource-efficient neural structure. The proposed model estimates a binary representation of weights by taking into account the approximation error with an additional term. A novel regularization term is introduced that is suitable for all threshold-based binary precision networks.
  arXiv  Detail & Related papers  (2021-05-09T10:11:26Z)
• Learning Frequency Domain Approximation for Binary Neural Networks [68.79904499480025]
  We propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2021-03-01T08:25:26Z)
• Binary Graph Neural Networks [69.51765073772226]
  Graph Neural Networks (GNNs) have emerged as a powerful and flexible framework for representation learning on irregular data. In this paper, we present and evaluate different strategies for the binarization of graph neural networks. We show that through careful design of the models, and control of the training process, binary graph neural networks can be trained at only a moderate cost in accuracy on challenging benchmarks.
  arXiv  Detail & Related papers  (2020-12-31T18:48:58Z)
• Bi-GCN: Binary Graph Convolutional Network [57.733849700089955]
  We propose a Binary Graph Convolutional Network (Bi-GCN), which binarizes both the network parameters and input node features. Our Bi-GCN can reduce the memory consumption by an average of 30x for both the network parameters and input data, and accelerate the inference speed by an average of 47x.
  arXiv  Detail & Related papers  (2020-10-15T07:26:23Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• Training Binary Neural Networks with Real-to-Binary Convolutions [52.91164959767517]
  We show how to train binary networks to within a few percent points of the full precision counterpart. We show how to build a strong baseline, which already achieves state-of-the-art accuracy. We show that, when putting all of our improvements together, the proposed model beats the current state of the art by more than 5% top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2020-03-25T17:54:38Z)
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations [16.92918746295432]
  We propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.
  arXiv  Detail & Related papers  (2020-02-16T06:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.