IC Neuron: An Efficient Unit to Construct Neural Networks
- URL: http://arxiv.org/abs/2011.11271v1
- Date: Mon, 23 Nov 2020 08:36:48 GMT
- Title: IC Neuron: An Efficient Unit to Construct Neural Networks
- Authors: Junyi An, Fengshan Liu, Jian Zhao and Furao Shen
- Abstract summary: We propose a new neuron model that can represent more complex distributions.
The Inter-layer collision (IC) neuron divides the input space into multiple subspaces used to represent different linear transformations.
We build the IC networks by integrating the IC neurons into the fully-connected (FC), convolutional, and recurrent structures.
- Score: 8.926478245654703
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: As a popular machine learning method, neural networks can be used to solve
many complex tasks. Their strong generalization ability comes from the
representation ability of the basic neuron model. The most popular neuron is
the MP neuron, which uses a linear transformation and a non-linear activation
function to process the input successively. Inspired by the elastic collision
model in physics, we propose a new neuron model that can represent more complex
distributions. We term it Inter-layer collision (IC) neuron. The IC neuron
divides the input space into multiple subspaces used to represent different
linear transformations. This operation enhanced non-linear representation
ability and emphasizes some useful input features for the given task. We build
the IC networks by integrating the IC neurons into the fully-connected (FC),
convolutional, and recurrent structures. The IC networks outperform the
traditional networks in a wide range of experiments. We believe that the IC
neuron can be a basic unit to build network structures.
Related papers
- Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks [0.0]
We introduce and evaluate a brain-like neural network model capable of unsupervised representation learning.
The model was tested on a diverse set of popular machine learning benchmarks.
arXiv Detail & Related papers (2024-06-07T08:32:30Z) - Single Neuromorphic Memristor closely Emulates Multiple Synaptic
Mechanisms for Energy Efficient Neural Networks [71.79257685917058]
We demonstrate memristive nano-devices based on SrTiO3 that inherently emulate all these synaptic functions.
These memristors operate in a non-filamentary, low conductance regime, which enables stable and energy efficient operation.
arXiv Detail & Related papers (2024-02-26T15:01:54Z) - Expressivity of Spiking Neural Networks [15.181458163440634]
We study the capabilities of spiking neural networks where information is encoded in the firing time of neurons.
In contrast to ReLU networks, we prove that spiking neural networks can realize both continuous and discontinuous functions.
arXiv Detail & Related papers (2023-08-16T08:45:53Z) - Neural network with optimal neuron activation functions based on
additive Gaussian process regression [0.0]
More flexible neuron activation functions would allow using fewer neurons and layers and improve expressive power.
We show that additive Gaussian process regression (GPR) can be used to construct optimal neuron activation functions that are individual to each neuron.
An approach is also introduced that avoids non-linear fitting of neural network parameters.
arXiv Detail & Related papers (2023-01-13T14:19:17Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - POPPINS : A Population-Based Digital Spiking Neuromorphic Processor with
Integer Quadratic Integrate-and-Fire Neurons [50.591267188664666]
We propose a population-based digital spiking neuromorphic processor in 180nm process technology with two hierarchy populations.
The proposed approach enables the developments of biomimetic neuromorphic system and various low-power, and low-latency inference processing applications.
arXiv Detail & Related papers (2022-01-19T09:26:34Z) - Continual Learning with Deep Artificial Neurons [0.0]
We introduce Deep Artificial Neurons (DANs), which are themselves realized as deep neural networks.
We demonstrate that it is possible to meta-learn a single parameter vector, which we dub a neuronal phenotype, shared by all DANs in the network.
We show that a suitable neuronal phenotype can endow a single network with an innate ability to update its synapses with minimal forgetting.
arXiv Detail & Related papers (2020-11-13T17:50:10Z) - Stability of Algebraic Neural Networks to Small Perturbations [179.55535781816343]
Algebraic neural networks (AlgNNs) are composed of a cascade of layers each one associated to and algebraic signal model.
We show how any architecture that uses a formal notion of convolution can be stable beyond particular choices of the shift operator.
arXiv Detail & Related papers (2020-10-22T09:10:16Z) - Flexible Transmitter Network [84.90891046882213]
Current neural networks are mostly built upon the MP model, which usually formulates the neuron as executing an activation function on the real-valued weighted aggregation of signals received from other neurons.
We propose the Flexible Transmitter (FT) model, a novel bio-plausible neuron model with flexible synaptic plasticity.
We present the Flexible Transmitter Network (FTNet), which is built on the most common fully-connected feed-forward architecture.
arXiv Detail & Related papers (2020-04-08T06:55:12Z) - Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy.
We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
arXiv Detail & Related papers (2020-02-02T21:09:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.