Self-Organized Operational Neural Networks with Generative Neurons

• URL: http://arxiv.org/abs/2004.11778v1
• Date: Fri, 24 Apr 2020 14:37:56 GMT
• Title: Self-Organized Operational Neural Networks with Generative Neurons
• Authors: Serkan Kiranyaz, Junaid Malik, Habib Ben Abdallah, Turker Ince, Alexandros Iosifidis and Moncef Gabbouj
• Abstract summary: ONNs are heterogenous networks with a generalized neuron model that can encapsulate any set of non-linear operators. We propose Self-organized ONNs (Self-ONNs) with generative neurons that have the ability to adapt (optimize) the nodal operator of each connection.
• Score: 87.32169414230822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Operational Neural Networks (ONNs) have recently been proposed to address the well-known limitations and drawbacks of conventional Convolutional Neural Networks (CNNs) such as network homogeneity with the sole linear neuron model. ONNs are heterogenous networks with a generalized neuron model that can encapsulate any set of non-linear operators to boost diversity and to learn highly complex and multi-modal functions or spaces with minimal network complexity and training data. However, Greedy Iterative Search (GIS) method, which is the search method used to find optimal operators in ONNs takes many training sessions to find a single operator set per layer. This is not only computationally demanding, but the network heterogeneity is also limited since the same set of operators will then be used for all neurons in each layer. Moreover, the performance of ONNs directly depends on the operator set library used, which introduces a certain risk of performance degradation especially when the optimal operator set required for a particular task is missing from the library. In order to address these issues and achieve an ultimate heterogeneity level to boost the network diversity along with computational efficiency, in this study we propose Self-organized ONNs (Self-ONNs) with generative neurons that have the ability to adapt (optimize) the nodal operator of each connection during the training process. Therefore, Self-ONNs can have an utmost heterogeneity level required by the learning problem at hand. Moreover, this ability voids the need of having a fixed operator set library and the prior operator search within the library in order to find the best possible set of operators. We further formulate the training method to back-propagate the error through the operational layers of Self-ONNs.

Related papers

• Fully Spiking Actor Network with Intra-layer Connections for Reinforcement Learning [51.386945803485084]
  We focus on the task where the agent needs to learn multi-dimensional deterministic policies to control. Most existing spike-based RL methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully-connected layer. To develop a fully spiking actor network without any floating-point matrix operations, we draw inspiration from the non-spiking interneurons found in insects.
  arXiv  Detail & Related papers  (2024-01-09T07:31:34Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Nonlocal Kernel Network (NKN): a Stable and Resolution-Independent Deep Neural Network [23.465930256410722]
  Nonlocal kernel network (NKN) is resolution independent, characterized by deep neural networks. NKN is capable of handling a variety of tasks such as learning governing equations and classifying images.
  arXiv  Detail & Related papers  (2022-01-06T19:19:35Z)
• Efficient and Robust Mixed-Integer Optimization Methods for Training Binarized Deep Neural Networks [0.07614628596146598]
  We study deep neural networks with binary activation functions and continuous or integer weights (BDNN) We show that the BDNN can be reformulated as a mixed-integer linear program with bounded weight space which can be solved to global optimality by classical mixed-integer programming solvers. For the first time a robust model is presented which enforces robustness of the BDNN during training.
  arXiv  Detail & Related papers  (2021-10-21T18:02:58Z)
• Neural Operator: Learning Maps Between Function Spaces [75.93843876663128]
  We propose a generalization of neural networks to learn operators, termed neural operators, that map between infinite dimensional function spaces. We prove a universal approximation theorem for our proposed neural operator, showing that it can approximate any given nonlinear continuous operator. An important application for neural operators is learning surrogate maps for the solution operators of partial differential equations.
  arXiv  Detail & Related papers  (2021-08-19T03:56:49Z)
• Learning Power Control for Cellular Systems with Heterogeneous Graph Neural Network [37.060397377445504]
  We show that the power control policy has a combination of different PI and PE properties, and existing HetGNN does not satisfy these properties. We design a parameter sharing scheme for HetGNN such that the learned relationship satisfies the desired properties.
  arXiv  Detail & Related papers  (2020-11-06T02:41:38Z)
• Exploiting Heterogeneity in Operational Neural Networks by Synaptic Plasticity [87.32169414230822]
  Recently proposed network model, Operational Neural Networks (ONNs), can generalize the conventional Convolutional Neural Networks (CNNs) In this study the focus is drawn on searching the best-possible operator set(s) for the hidden neurons of the network based on the Synaptic Plasticity paradigm that poses the essential learning theory in biological neurons. Experimental results over highly challenging problems demonstrate that the elite ONNs even with few neurons and layers can achieve a superior learning performance than GIS-based ONNs.
  arXiv  Detail & Related papers  (2020-08-21T19:03:23Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)
• FastONN -- Python based open-source GPU implementation for Operational Neural Networks [25.838282412957675]
  This work introduces a fast GPU-enabled library for training operational neural networks, FastONN. FastONN is based on a novel vectorized formulation of the operational neurons. bundled auxiliary modules offer interfaces for performance tracking and checkpointing across different data partitions and customized metrics.
  arXiv  Detail & Related papers  (2020-06-03T13:33:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.