Super Neurons

• URL: http://arxiv.org/abs/2109.01594v2
• Date: Sat, 15 Apr 2023 21:14:09 GMT
• Title: Super Neurons
• Authors: Serkan Kiranyaz, Junaid Malik, Mehmet Yamac, Mert Duman, Ilke Adalioglu, Esin Guldogan, Turker Ince, and Moncef Gabbouj
• Abstract summary: Self-Organized Operational Neural Networks (Self-ONNs) have been proposed as new-generation neural network models with nonlinear learning units. Self-ONNs have a common drawback: localized (fixed) kernel operations. This article presents superior (generative) neuron models that allow random or learnable kernel shifts.
• Score: 18.710336981941147
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-Organized Operational Neural Networks (Self-ONNs) have recently been proposed as new-generation neural network models with nonlinear learning units, i.e., the generative neurons that yield an elegant level of diversity; however, like its predecessor, conventional Convolutional Neural Networks (CNNs), they still have a common drawback: localized (fixed) kernel operations. This severely limits the receptive field and information flow between layers and thus brings the necessity for deep and complex models. It is highly desired to improve the receptive field size without increasing the kernel dimensions. This requires a significant upgrade over the generative neurons to achieve the non-localized kernel operations for each connection between consecutive layers. In this article, we present superior (generative) neuron models (or super neurons in short) that allow random or learnable kernel shifts and thus can increase the receptive field size of each connection. The kernel localization process varies among the two super-neuron models. The first model assumes randomly localized kernels within a range and the second one learns (optimizes) the kernel locations during training. An extensive set of comparative evaluations against conventional and deformable convolutional, along with the generative neurons demonstrates that super neurons can empower Self-ONNs to achieve a superior learning and generalization capability with a minimal computational complexity burden.

Related papers

• Novel Kernel Models and Exact Representor Theory for Neural Networks Beyond the Over-Parameterized Regime [52.00917519626559]
  This paper presents two models of neural-networks and their training applicable to neural networks of arbitrary width, depth and topology. We also present an exact novel representor theory for layer-wise neural network training with unregularized gradient descent in terms of a local-extrinsic neural kernel (LeNK) This representor theory gives insight into the role of higher-order statistics in neural network training and the effect of kernel evolution in neural-network kernel models.
  arXiv  Detail & Related papers  (2024-05-24T06:30:36Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Why Quantization Improves Generalization: NTK of Binary Weight Neural Networks [33.08636537654596]
  We take the binary weights in a neural network as random variables under rounding, and study the distribution propagation over different layers in the neural network. We propose a quasi neural network to approximate the distribution propagation, which is a neural network with continuous parameters and smooth activation function.
  arXiv  Detail & Related papers  (2022-06-13T06:11:21Z)
• Deep Learning in Random Neural Fields: Numerical Experiments via Neural Tangent Kernel [10.578941575914516]
  A biological neural network in the cortex forms a neural field. Neurons in the field have their own receptive fields, and connection weights between two neurons are random but highly correlated when they are in close proximity in receptive fields. We show that such a multilayer neural field is more robust than conventional models when input patterns are deformed by noise disturbances.
  arXiv  Detail & Related papers  (2022-02-10T18:57:10Z)
• POPPINS : A Population-Based Digital Spiking Neuromorphic Processor with Integer Quadratic Integrate-and-Fire Neurons [50.591267188664666]
  We propose a population-based digital spiking neuromorphic processor in 180nm process technology with two hierarchy populations. The proposed approach enables the developments of biomimetic neuromorphic system and various low-power, and low-latency inference processing applications.
  arXiv  Detail & Related papers  (2022-01-19T09:26:34Z)
• Improving Spiking Neural Network Accuracy Using Time-based Neurons [0.24366811507669117]
  Research on neuromorphic computing systems based on low-power spiking neural networks using analog neurons is in the spotlight. As technology scales down, analog neurons are difficult to scale, and they suffer from reduced voltage headroom/dynamic range and circuit nonlinearities. This paper first models the nonlinear behavior of existing current-mirror-based voltage-domain neurons designed in a 28nm process, and show SNN inference accuracy can be severely degraded by the effect of neuron's nonlinearity. We propose a novel neuron, which processes incoming spikes in the time domain and greatly improves the linearity, thereby improving the inference accuracy compared to the
  arXiv  Detail & Related papers  (2022-01-05T00:24:45Z)
• The Separation Capacity of Random Neural Networks [78.25060223808936]
  We show that a sufficiently large two-layer ReLU-network with standard Gaussian weights and uniformly distributed biases can solve this problem with high probability. We quantify the relevant structure of the data in terms of a novel notion of mutual complexity.
  arXiv  Detail & Related papers  (2021-07-31T10:25:26Z)
• Continual Learning with Deep Artificial Neurons [0.0]
  We introduce Deep Artificial Neurons (DANs), which are themselves realized as deep neural networks. We demonstrate that it is possible to meta-learn a single parameter vector, which we dub a neuronal phenotype, shared by all DANs in the network. We show that a suitable neuronal phenotype can endow a single network with an innate ability to update its synapses with minimal forgetting.
  arXiv  Detail & Related papers  (2020-11-13T17:50:10Z)
• Exploiting Heterogeneity in Operational Neural Networks by Synaptic Plasticity [87.32169414230822]
  Recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) In this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the Synaptic Plasticity paradigm that poses the essential learning theory in biological neurons. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs.
  arXiv  Detail & Related papers  (2020-08-21T19:03:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.