Designed Dithering Sign Activation for Binary Neural Networks

• URL: http://arxiv.org/abs/2405.02220v2
• Date: Thu, 9 May 2024 16:02:12 GMT
• Title: Designed Dithering Sign Activation for Binary Neural Networks
• Authors: Brayan Monroy, Juan Estupiñan, Tatiana Gelvez-Barrera, Jorge Bacca, Henry Arguello,
• Abstract summary: This work proposes an activation that applies multiple thresholds following dithering principles, shifting the Sign activation function for each pixel according to a spatially periodic threshold kernel. Experiments over the classification task demonstrate the effectiveness of the designed dithering Sign activation function as an alternative activation for binary neural networks, without increasing the computational cost.
• Score: 15.087814338685968
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Binary Neural Networks emerged as a cost-effective and energy-efficient solution for computer vision tasks by binarizing either network weights or activations. However, common binary activations, such as the Sign activation function, abruptly binarize the values with a single threshold, losing fine-grained details in the feature outputs. This work proposes an activation that applies multiple thresholds following dithering principles, shifting the Sign activation function for each pixel according to a spatially periodic threshold kernel. Unlike literature methods, the shifting is defined jointly for a set of adjacent pixels, taking advantage of spatial correlations. Experiments over the classification task demonstrate the effectiveness of the designed dithering Sign activation function as an alternative activation for binary neural networks, without increasing the computational cost. Further, DeSign balances the preservation of details with the efficiency of binary operations.

Related papers

• Neuromorphic Synergy for Video Binarization [54.195375576583864]
  Bimodal objects serve as a visual form to embed information that can be easily recognized by vision systems. Neuromorphic cameras offer new capabilities for alleviating motion blur, but it is non-trivial to first de-blur and then binarize the images in a real-time manner. We propose an event-based binary reconstruction method that leverages the prior knowledge of the bimodal target's properties to perform inference independently in both event space and image space. We also develop an efficient integration method to propagate this binary image to high frame rate binary video.
  arXiv  Detail & Related papers  (2024-02-20T01:43:51Z)
• Efficient Representation of the Activation Space in Deep Neural Networks [5.224743522146324]
  We propose a model-agnostic framework for creating representations of activations in deep neural networks. The framework reduces memory usage by 30% with up to 4 times faster p-value computing time. As we do not persist raw data at inference time, we could potentially reduce susceptibility to attacks and privacy issues.
  arXiv  Detail & Related papers  (2023-12-13T13:46:14Z)
• Fractional Concepts in Neural Networks: Enhancing Activation and Loss Functions [0.7614628596146602]
  The paper presents a method for using fractional concepts in a neural network to modify the activation and loss functions. This will enable neurons in the network to adjust their activation functions to match input data better and reduce output errors.
  arXiv  Detail & Related papers  (2023-10-18T10:49:29Z)
• Multilayer Multiset Neuronal Networks -- MMNNs [55.2480439325792]
  The present work describes multilayer multiset neuronal networks incorporating two or more layers of coincidence similarity neurons. The work also explores the utilization of counter-prototype points, which are assigned to the image regions to be avoided.
  arXiv  Detail & Related papers  (2023-08-28T12:55:13Z)
• STL: A Signed and Truncated Logarithm Activation Function for Neural Networks [5.9622541907827875]
  Activation functions play an essential role in neural networks. We present a novel signed and truncated logarithm function as activation function. The suggested activation function can be applied in a large range of neural networks.
  arXiv  Detail & Related papers  (2023-07-31T03:41:14Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training [68.63354877166756]
  ActNN is a memory-efficient training framework that stores randomly quantized activations for back propagation. ActNN reduces the memory footprint of the activation by 12x, and it enables training with a 6.6x to 14x larger batch size.
  arXiv  Detail & Related papers  (2021-04-29T05:50:54Z)
• Training Binary Neural Networks through Learning with Noisy Supervision [76.26677550127656]
  This paper formalizes the binarization operations over neural networks from a learning perspective. Experimental results on benchmark datasets indicate that the proposed binarization technique attains consistent improvements over baselines.
  arXiv  Detail & Related papers  (2020-10-10T01:59:39Z)
• BiDet: An Efficient Binarized Object Detector [96.19708396510894]
  We propose a binarized neural network learning method called BiDet for efficient object detection. Our BiDet fully utilizes the representational capacity of the binary neural networks for object detection by redundancy removal. Our method outperforms the state-of-the-art binary neural networks by a sizable margin.
  arXiv  Detail & Related papers  (2020-03-09T08:16:16Z)
• BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activations [16.92918746295432]
  We propose a new training scheme for binary activation networks called BinaryDuo in which two binary activations are coupled into a ternary activation during training. Experimental results show that BinaryDuo outperforms state-of-the-art BNNs on various benchmarks with the same amount of parameters and computing cost.
  arXiv  Detail & Related papers  (2020-02-16T06:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.