Correlative Information Maximization: A Biologically Plausible Approach
to Supervised Deep Neural Networks without Weight Symmetry
- URL: http://arxiv.org/abs/2306.04810v3
- Date: Tue, 17 Oct 2023 11:32:52 GMT
- Title: Correlative Information Maximization: A Biologically Plausible Approach
to Supervised Deep Neural Networks without Weight Symmetry
- Authors: Bariscan Bozkurt, Cengiz Pehlevan, Alper T Erdogan
- Abstract summary: We propose a new normative approach to describe the signal propagation in biological neural networks in both forward and backward directions.
This framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm.
Our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths.
- Score: 43.584567991256925
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The backpropagation algorithm has experienced remarkable success in training
large-scale artificial neural networks; however, its biological plausibility
has been strongly criticized, and it remains an open question whether the brain
employs supervised learning mechanisms akin to it. Here, we propose correlative
information maximization between layer activations as an alternative normative
approach to describe the signal propagation in biological neural networks in
both forward and backward directions. This new framework addresses many
concerns about the biological-plausibility of conventional artificial neural
networks and the backpropagation algorithm. The coordinate descent-based
optimization of the corresponding objective, combined with the mean square
error loss function for fitting labeled supervision data, gives rise to a
neural network structure that emulates a more biologically realistic network of
multi-compartment pyramidal neurons with dendritic processing and lateral
inhibitory neurons. Furthermore, our approach provides a natural resolution to
the weight symmetry problem between forward and backward signal propagation
paths, a significant critique against the plausibility of the conventional
backpropagation algorithm. This is achieved by leveraging two alternative, yet
equivalent forms of the correlative mutual information objective. These
alternatives intrinsically lead to forward and backward prediction networks
without weight symmetry issues, providing a compelling solution to this
long-standing challenge.
Related papers
- Deep Learning without Weight Symmetry [1.3812010983144802]
Backpropagation (BP) is a foundational algorithm for training artificial neural networks.
BP is often considered biologically implausible.
Here we introduce the Product Feedback Alignment (PFA) algorithm.
arXiv Detail & Related papers (2024-05-31T03:11:19Z) - Seeking Next Layer Neurons' Attention for Error-Backpropagation-Like
Training in a Multi-Agent Network Framework [6.446189857311325]
We propose a local objective for neurons that align them to exhibit similarities to error-backpropagation.
We examine a neural network comprising decentralized, self-interested neurons seeking to maximize their local objective.
We demonstrate the learning capacity of these multi-agent neural networks through experiments on three datasets.
arXiv Detail & Related papers (2023-10-15T21:07:09Z) - Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence.
We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers.
This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
arXiv Detail & Related papers (2023-07-20T13:34:11Z) - Cycle Consistency-based Uncertainty Quantification of Neural Networks in
Inverse Imaging Problems [10.992084413881592]
Uncertainty estimation is critical for numerous applications of deep neural networks.
We show an uncertainty quantification approach for deep neural networks used in inverse problems based on cycle consistency.
arXiv Detail & Related papers (2023-05-22T09:23:18Z) - Contrastive-Signal-Dependent Plasticity: Self-Supervised Learning in Spiking Neural Circuits [61.94533459151743]
This work addresses the challenge of designing neurobiologically-motivated schemes for adjusting the synapses of spiking networks.
Our experimental simulations demonstrate a consistent advantage over other biologically-plausible approaches when training recurrent spiking networks.
arXiv Detail & Related papers (2023-03-30T02:40:28Z) - Convergence Guarantees of Overparametrized Wide Deep Inverse Prior [1.5362025549031046]
Inverse Priors is an unsupervised approach to transform a random input into an object whose image under the forward model matches the observation.
We provide overparametrization bounds under which such network trained via continuous-time gradient descent will converge exponentially fast with high probability.
This work is thus a first step towards a theoretical understanding of overparametrized DIP networks, and more broadly it participates to the theoretical understanding of neural networks in inverse problem settings.
arXiv Detail & Related papers (2023-03-20T16:49:40Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
We propose a topological perspective to represent a network into a complete graph for analysis.
By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner.
This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
arXiv Detail & Related papers (2020-08-19T04:53:31Z) - Predictive coding in balanced neural networks with noise, chaos and
delays [24.76770648963407]
We introduce an analytically tractable model of balanced predictive coding, in which the degree of balance and the degree of weight disorder can be dissociated.
Our work provides and solves a general theoretical framework for dissecting the differential contributions neural noise, synaptic disorder, chaos, synaptic delays, and balance to the fidelity of predictive neural codes.
arXiv Detail & Related papers (2020-06-25T05:03:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.