Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry

• URL: http://arxiv.org/abs/2306.04810v3
• Date: Tue, 17 Oct 2023 11:32:52 GMT
• Title: Correlative Information Maximization: A Biologically Plausible Approach to Supervised Deep Neural Networks without Weight Symmetry
• Authors: Bariscan Bozkurt, Cengiz Pehlevan, Alper T Erdogan
• Abstract summary: We propose a new normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. Our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths.
• Score: 43.584567991256925
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The backpropagation algorithm has experienced remarkable success in training large-scale artificial neural networks; however, its biological plausibility has been strongly criticized, and it remains an open question whether the brain employs supervised learning mechanisms akin to it. Here, we propose correlative information maximization between layer activations as an alternative normative approach to describe the signal propagation in biological neural networks in both forward and backward directions. This new framework addresses many concerns about the biological-plausibility of conventional artificial neural networks and the backpropagation algorithm. The coordinate descent-based optimization of the corresponding objective, combined with the mean square error loss function for fitting labeled supervision data, gives rise to a neural network structure that emulates a more biologically realistic network of multi-compartment pyramidal neurons with dendritic processing and lateral inhibitory neurons. Furthermore, our approach provides a natural resolution to the weight symmetry problem between forward and backward signal propagation paths, a significant critique against the plausibility of the conventional backpropagation algorithm. This is achieved by leveraging two alternative, yet equivalent forms of the correlative mutual information objective. These alternatives intrinsically lead to forward and backward prediction networks without weight symmetry issues, providing a compelling solution to this long-standing challenge.

Related papers

• Deep Learning without Weight Symmetry [1.3812010983144802]
  Backpropagation (BP) is a foundational algorithm for training artificial neural networks. BP is often considered biologically implausible. Here we introduce the Product Feedback Alignment (PFA) algorithm.
  arXiv  Detail & Related papers  (2024-05-31T03:11:19Z)
• Seeking Next Layer Neurons' Attention for Error-Backpropagation-Like Training in a Multi-Agent Network Framework [6.446189857311325]
  We propose a local objective for neurons that align them to exhibit similarities to error-backpropagation. We examine a neural network comprising decentralized, self-interested neurons seeking to maximize their local objective. We demonstrate the learning capacity of these multi-agent neural networks through experiments on three datasets.
  arXiv  Detail & Related papers  (2023-10-15T21:07:09Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Cycle Consistency-based Uncertainty Quantification of Neural Networks in Inverse Imaging Problems [10.992084413881592]
  Uncertainty estimation is critical for numerous applications of deep neural networks. We show an uncertainty quantification approach for deep neural networks used in inverse problems based on cycle consistency.
  arXiv  Detail & Related papers  (2023-05-22T09:23:18Z)
• Convergence Guarantees of Overparametrized Wide Deep Inverse Prior [1.5362025549031046]
  Inverse Priors is an unsupervised approach to transform a random input into an object whose image under the forward model matches the observation. We provide overparametrization bounds under which such network trained via continuous-time gradient descent will converge exponentially fast with high probability. This work is thus a first step towards a theoretical understanding of overparametrized DIP networks, and more broadly it participates to the theoretical understanding of neural networks in inverse problem settings.
  arXiv  Detail & Related papers  (2023-03-20T16:49:40Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• Predictive coding in balanced neural networks with noise, chaos and delays [24.76770648963407]
  We introduce an analytically tractable model of balanced predictive coding, in which the degree of balance and the degree of weight disorder can be dissociated. Our work provides and solves a general theoretical framework for dissecting the differential contributions neural noise, synaptic disorder, chaos, synaptic delays, and balance to the fidelity of predictive neural codes.
  arXiv  Detail & Related papers  (2020-06-25T05:03:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.