Computational and Storage Efficient Quadratic Neurons for Deep Neural Networks

• URL: http://arxiv.org/abs/2306.07294v2
• Date: Mon, 27 Nov 2023 10:21:24 GMT
• Title: Computational and Storage Efficient Quadratic Neurons for Deep Neural Networks
• Authors: Chuangtao Chen and Grace Li Zhang and Xunzhao Yin and Cheng Zhuo and Ulf Schlichtmann and Bing Li
• Abstract summary: Experimental results have demonstrated that the proposed quadratic neuron structure exhibits superior computational and storage efficiency across various tasks. This work introduces an efficient quadratic neuron architecture distinguished by its enhanced utilization of second-order computational information.
• Score: 10.379191500493503
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deep neural networks (DNNs) have been widely deployed across diverse domains such as computer vision and natural language processing. However, the impressive accomplishments of DNNs have been realized alongside extensive computational demands, thereby impeding their applicability on resource-constrained devices. To address this challenge, many researchers have been focusing on basic neuron structures, the fundamental building blocks of neural networks, to alleviate the computational and storage cost. In this work, an efficient quadratic neuron architecture distinguished by its enhanced utilization of second-order computational information is introduced. By virtue of their better expressivity, DNNs employing the proposed quadratic neurons can attain similar accuracy with fewer neurons and computational cost. Experimental results have demonstrated that the proposed quadratic neuron structure exhibits superior computational and storage efficiency across various tasks when compared with both linear and non-linear neurons in prior work.

Related papers

• Enhancing learning in spiking neural networks through neuronal heterogeneity and neuromodulatory signaling [52.06722364186432]
  We propose a biologically-informed framework for enhancing artificial neural networks (ANNs) Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors. We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
  arXiv  Detail & Related papers  (2024-07-05T14:11:28Z)
• Automated Natural Language Explanation of Deep Visual Neurons with Large Models [43.178568768100305]
  This paper proposes a novel post-hoc framework for generating semantic explanations of neurons with large foundation models. Our framework is designed to be compatible with various model architectures and datasets, automated and scalable neuron interpretation.
  arXiv  Detail & Related papers  (2023-10-16T17:04:51Z)
• To Spike or Not To Spike: A Digital Hardware Perspective on Deep Learning Acceleration [4.712922151067433]
  As deep learning models scale, they become increasingly competitive from domains spanning from computer vision to natural language processing. The power efficiency of the biological brain outperforms any large-scale deep learning ( DL ) model. Neuromorphic computing tries to mimic the brain operations to improve the efficiency of DL models.
  arXiv  Detail & Related papers  (2023-06-27T19:04:00Z)
• A Hybrid Neural Coding Approach for Pattern Recognition with Spiking Neural Networks [53.31941519245432]
  Brain-inspired spiking neural networks (SNNs) have demonstrated promising capabilities in solving pattern recognition tasks. These SNNs are grounded on homogeneous neurons that utilize a uniform neural coding for information representation. In this study, we argue that SNN architectures should be holistically designed to incorporate heterogeneous coding schemes.
  arXiv  Detail & Related papers  (2023-05-26T02:52:12Z)
• Exploiting Noise as a Resource for Computation and Learning in Spiking Neural Networks [32.0086664373154]
  This study introduces the noisy spiking neural network (NSNN) and the noise-driven learning rule (NDL) NSNN provides a theoretical framework that yields scalable, flexible, and reliable computation.
  arXiv  Detail & Related papers  (2023-05-25T13:21:26Z)
• Efficient, probabilistic analysis of combinatorial neural codes [0.0]
  neural networks encode inputs in the form of combinations of individual neurons' activities. These neural codes present a computational challenge due to their high dimensionality and often large volumes of data. We apply methods previously applied to small examples and apply them to large neural codes generated by experiments.
  arXiv  Detail & Related papers  (2022-10-19T11:58:26Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Extrapolation and Spectral Bias of Neural Nets with Hadamard Product: a Polynomial Net Study [55.12108376616355]
  The study on NTK has been devoted to typical neural network architectures, but is incomplete for neural networks with Hadamard products (NNs-Hp) In this work, we derive the finite-width-K formulation for a special class of NNs-Hp, i.e., neural networks. We prove their equivalence to the kernel regression predictor with the associated NTK, which expands the application scope of NTK.
  arXiv  Detail & Related papers  (2022-09-16T06:36:06Z)
• Effective and Efficient Computation with Multiple-timescale Spiking Recurrent Neural Networks [0.9790524827475205]
  We show how a novel type of adaptive spiking recurrent neural network (SRNN) is able to achieve state-of-the-art performance. We calculate a $>$100x energy improvement for our SRNNs over classical RNNs on the harder tasks.
  arXiv  Detail & Related papers  (2020-05-24T01:04:53Z)
• Spiking Neural Networks Hardware Implementations and Challenges: a Survey [53.429871539789445]
  Spiking Neural Networks are cognitive algorithms mimicking neuron and synapse operational principles. We present the state of the art of hardware implementations of spiking neural networks. We discuss the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level.
  arXiv  Detail & Related papers  (2020-05-04T13:24:00Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.