Constrained Parameter Inference as a Principle for Learning

• URL: http://arxiv.org/abs/2203.13203v1
• Date: Tue, 22 Mar 2022 13:40:57 GMT
• Title: Constrained Parameter Inference as a Principle for Learning
• Authors: Nasir Ahmad, Ellen Schrader, Marcel van Gerven
• Abstract summary: We propose constrained parameter inference (COPI) as a new principle for learning. COPI allows for the estimation of network parameters under the constraints of decorrelated neural inputs and top-down perturbations of neural states. We show that COPI not only is more biologically plausible but also provides distinct advantages for fast learning, compared with standard backpropagation of error.
• Score: 5.080518039966762
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning in biological and artificial neural networks is often framed as a problem in which targeted error signals guide parameter updating for more optimal network behaviour. Backpropagation of error (BP) is an example of such an approach and has proven to be a highly successful application of stochastic gradient descent to deep neural networks. However, BP relies on the global transmission of gradient information and has therefore been criticised for its biological implausibility. We propose constrained parameter inference (COPI) as a new principle for learning. COPI allows for the estimation of network parameters under the constraints of decorrelated neural inputs and top-down perturbations of neural states. We show that COPI not only is more biologically plausible but also provides distinct advantages for fast learning, compared with standard backpropagation of error.

Related papers

• DFA-GNN: Forward Learning of Graph Neural Networks by Direct Feedback Alignment [57.62885438406724]
  Graph neural networks are recognized for their strong performance across various applications. BP has limitations that challenge its biological plausibility and affect the efficiency, scalability and parallelism of training neural networks for graph-based tasks. We propose DFA-GNN, a novel forward learning framework tailored for GNNs with a case study of semi-supervised learning.
  arXiv  Detail & Related papers  (2024-06-04T07:24:51Z)
• Evolutionary algorithms as an alternative to backpropagation for supervised training of Biophysical Neural Networks and Neural ODEs [12.357635939839696]
  We investigate the use of "gradient-estimating" evolutionary algorithms for training biophysically based neural networks. We find that EAs have several advantages making them desirable over direct BP. Our findings suggest that biophysical neurons could provide useful benchmarks for testing the limits of BP methods.
  arXiv  Detail & Related papers  (2023-11-17T20:59:57Z)
• Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry [43.584567991256925]
  We propose a new normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. Our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths.
  arXiv  Detail & Related papers  (2023-06-07T22:14:33Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Learning to Precode for Integrated Sensing and Communications Systems [11.689567114100514]
  We present an unsupervised learning neural model to design transmit precoders for ISAC systems. We show that the proposed method outperforms traditional optimization-based methods in presence of channel estimation errors.
  arXiv  Detail & Related papers  (2023-03-11T11:24:18Z)
• Towards Evaluating and Training Verifiably Robust Neural Networks [81.39994285743555]
  We study the relationship between IBP and CROWN, and prove that CROWN is always tighter than IBP when choosing appropriate bounding lines. We propose a relaxed version of CROWN, linear bound propagation (LBP), that can be used to verify large networks to obtain lower verified errors.
  arXiv  Detail & Related papers  (2021-04-01T13:03:48Z)
• Training Sparse Neural Network by Constraining Synaptic Weight on Unit Lp Sphere [2.429910016019183]
  constraining the synaptic weights on unit Lp-sphere enables the flexibly control of the sparsity with p. Our approach is validated by experiments on benchmark datasets covering a wide range of domains.
  arXiv  Detail & Related papers  (2021-03-30T01:02:31Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• A simple normative network approximates local non-Hebbian learning in the cortex [12.940770779756482]
  Neuroscience experiments demonstrate that the processing of sensory inputs by cortical neurons is modulated by instructive signals. Here, adopting a normative approach, we model these instructive signals as supervisory inputs guiding the projection of the feedforward data. Online algorithms can be implemented by neural networks whose synaptic learning rules resemble calcium plateau potential dependent plasticity observed in the cortex.
  arXiv  Detail & Related papers  (2020-10-23T20:49:44Z)
• Relaxing the Constraints on Predictive Coding Models [62.997667081978825]
  Predictive coding is an influential theory of cortical function which posits that the principal computation the brain performs is the minimization of prediction errors. Standard implementations of the algorithm still involve potentially neurally implausible features such as identical forward and backward weights, backward nonlinear derivatives, and 1-1 error unit connectivity. In this paper, we show that these features are not integral to the algorithm and can be removed either directly or through learning additional sets of parameters with Hebbian update rules without noticeable harm to learning performance.
  arXiv  Detail & Related papers  (2020-10-02T15:21:37Z)
• Rectified Linear Postsynaptic Potential Function for Backpropagation in Deep Spiking Neural Networks [55.0627904986664]
  Spiking Neural Networks (SNNs) usetemporal spike patterns to represent and transmit information, which is not only biologically realistic but also suitable for ultra-low-power event-driven neuromorphic implementation. This paper investigates the contribution of spike timing dynamics to information encoding, synaptic plasticity and decision making, providing a new perspective to design of future DeepSNNs and neuromorphic hardware systems.
  arXiv  Detail & Related papers  (2020-03-26T11:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.