Training Neural Networks Using the Property of Negative Feedback to Inverse a Function

• URL: http://arxiv.org/abs/2103.14115v1
• Date: Thu, 25 Mar 2021 20:13:53 GMT
• Title: Training Neural Networks Using the Property of Negative Feedback to Inverse a Function
• Authors: Md Munir Hasan and Jeremy Holleman
• Abstract summary: This paper describes how the property of a negative feedback system to perform inverse of a function can be used for training neural networks. We have applied this method to the MNIST dataset and obtained results that shows the method is viable for neural network training.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: With high forward gain, a negative feedback system has the ability to perform the inverse of a linear or non linear function that is in the feedback path. This property of negative feedback systems has been widely used in analog circuits to construct precise closed-loop functions. This paper describes how the property of a negative feedback system to perform inverse of a function can be used for training neural networks. This method does not require that the cost or activation functions be differentiable. Hence, it is able to learn a class of non-differentiable functions as well where a gradient descent-based method fails. We also show that gradient descent emerges as a special case of the proposed method. We have applied this method to the MNIST dataset and obtained results that shows the method is viable for neural network training. This method, to the best of our knowledge, is novel in machine learning.

Related papers

• Nonlinear functional regression by functional deep neural network with kernel embedding [20.306390874610635]
  We propose a functional deep neural network with an efficient and fully data-dependent dimension reduction method. The architecture of our functional net consists of a kernel embedding step, a projection step, and a deep ReLU neural network for the prediction. The utilization of smooth kernel embedding enables our functional net to be discretization invariant, efficient, and robust to noisy observations.
  arXiv  Detail & Related papers  (2024-01-05T16:43:39Z)
• Using Linear Regression for Iteratively Training Neural Networks [4.873362301533824]
  We present a simple linear regression based approach for learning the weights and biases of a neural network. The approach is intended to be to larger, more complex architectures.
  arXiv  Detail & Related papers  (2023-07-11T11:53:25Z)
• Koopman operator learning using invertible neural networks [0.6846628460229516]
  In Koopman operator theory, a finite-dimensional nonlinear system is transformed into an infinite but linear system using a set of observable functions. Current methodologies tend to disregard the importance of the invertibility of observable functions, which leads to inaccurate results. We propose FlowDMD, aka Flow-based Dynamic Mode Decomposition, that utilizes the Coupling Flow Invertible Neural Network (CF-INN) framework.
  arXiv  Detail & Related papers  (2023-06-30T04:26:46Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Theoretical Characterization of How Neural Network Pruning Affects its Generalization [131.1347309639727]
  This work makes the first attempt to study how different pruning fractions affect the model's gradient descent dynamics and generalization. It is shown that as long as the pruning fraction is below a certain threshold, gradient descent can drive the training loss toward zero. More surprisingly, the generalization bound gets better as the pruning fraction gets larger.
  arXiv  Detail & Related papers  (2023-01-01T03:10:45Z)
• The Forward-Forward Algorithm: Some Preliminary Investigations [91.3755431537592]
  The Forward-Forward algorithm replaces the forward and backward passes of backpropagation by two forward passes. If the positive and negative passes could be separated in time, the negative passes could be done offline.
  arXiv  Detail & Related papers  (2022-12-27T02:54:46Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Learning Linearized Assignment Flows for Image Labeling [70.540936204654]
  We introduce a novel algorithm for estimating optimal parameters of linearized assignment flows for image labeling. We show how to efficiently evaluate this formula using a Krylov subspace and a low-rank approximation.
  arXiv  Detail & Related papers  (2021-08-02T13:38:09Z)
• Learning Frequency Domain Approximation for Binary Neural Networks [68.79904499480025]
  We propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2021-03-01T08:25:26Z)
• A Use of Even Activation Functions in Neural Networks [0.35172332086962865]
  We propose an alternative approach to integrate existing knowledge or hypotheses of data structure by constructing custom activation functions. We show that using an even activation function in one of the fully connected layers improves neural network performance.
  arXiv  Detail & Related papers  (2020-11-23T20:33:13Z)
• Estimating Multiplicative Relations in Neural Networks [0.0]
  We will use properties of logarithmic functions to propose a pair of activation functions which can translate products into linear expression and learn using backpropagation. We will try to generalize this approach for some complex arithmetic functions and test the accuracy on a disjoint distribution with the training set.
  arXiv  Detail & Related papers  (2020-10-28T14:28:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.