A developmental approach for training deep belief networks
- URL: http://arxiv.org/abs/2207.05473v1
- Date: Tue, 12 Jul 2022 11:37:58 GMT
- Title: A developmental approach for training deep belief networks
- Authors: Matteo Zambra, Alberto Testolin, Michele De Filippo De Grazia, Marco
Zorzi
- Abstract summary: Deep belief networks (DBNs) are neural networks that can extract rich internal representations of the environment from the sensory data.
We present iDBN, an iterative learning algorithm for DBNs that allows to jointly update the connection weights across all layers of the hierarchy.
Our work paves the way to the use of iDBN for modeling neurocognitive development.
- Score: 0.46699574490885926
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Deep belief networks (DBNs) are stochastic neural networks that can extract
rich internal representations of the environment from the sensory data. DBNs
had a catalytic effect in triggering the deep learning revolution,
demonstrating for the very first time the feasibility of unsupervised learning
in networks with many layers of hidden neurons. Thanks to their biological and
cognitive plausibility, these hierarchical architectures have been also
successfully exploited to build computational models of human perception and
cognition in a variety of domains. However, learning in DBNs is usually carried
out in a greedy, layer-wise fashion, which does not allow to simulate the
holistic development of cortical circuits. Here we present iDBN, an iterative
learning algorithm for DBNs that allows to jointly update the connection
weights across all layers of the hierarchy. We test our algorithm on two
different sets of visual stimuli, and we show that network development can also
be tracked in terms of graph theoretical properties. DBNs trained using our
iterative approach achieve a final performance comparable to that of the greedy
counterparts, at the same time allowing to accurately analyze the gradual
development of internal representations in the generative model. Our work paves
the way to the use of iDBN for modeling neurocognitive development.
Related papers
- The Dynamic Net Architecture: Learning Robust and Holistic Visual Representations Through Self-Organizing Networks [3.9848584845601014]
We present a novel intelligent-system architecture called "Dynamic Net Architecture" (DNA)
DNA relies on recurrence-stabilized networks and discuss it in application to vision.
arXiv Detail & Related papers (2024-07-08T06:22:10Z) - Enhancing learning in artificial neural networks through cellular heterogeneity and neuromodulatory signaling [52.06722364186432]
We propose a biologically-informed framework for enhancing artificial neural networks (ANNs)
Our proposed dual-framework approach highlights the potential of spiking neural networks (SNNs) for emulating diverse spiking behaviors.
We outline how the proposed approach integrates brain-inspired compartmental models and task-driven SNNs, bioinspiration and complexity.
arXiv Detail & Related papers (2024-07-05T14:11:28Z) - Unsupervised representation learning with Hebbian synaptic and structural plasticity in brain-like feedforward neural networks [0.0]
We introduce and evaluate a brain-like neural network model capable of unsupervised representation learning.
The model was tested on a diverse set of popular machine learning benchmarks.
arXiv Detail & Related papers (2024-06-07T08:32:30Z) - Contrastive-Signal-Dependent Plasticity: Forward-Forward Learning of
Spiking Neural Systems [73.18020682258606]
We develop a neuro-mimetic architecture, composed of spiking neuronal units, where individual layers of neurons operate in parallel.
We propose an event-based generalization of forward-forward learning, which we call contrastive-signal-dependent plasticity (CSDP)
Our experimental results on several pattern datasets demonstrate that the CSDP process works well for training a dynamic recurrent spiking network capable of both classification and reconstruction.
arXiv Detail & Related papers (2023-03-30T02:40:28Z) - Identifying Equivalent Training Dynamics [3.793387630509845]
Topological conjugacy provides a precise definition of dynamical equivalence.
We develop a framework for identifying conjugate and non-conjugate training dynamics.
Our results illustrate the flexibility of our framework and highlight its potential for shedding new light on training dynamics.
arXiv Detail & Related papers (2023-02-17T22:15:20Z) - Developing hierarchical anticipations via neural network-based event
segmentation [14.059479351946386]
We model the development of hierarchical predictions via autonomously learned latent event codes.
We present a hierarchical recurrent neural network architecture, whose inductive learning biases foster the development of sparsely changing latent state.
A higher level network learns to predict the situations in which the latent states tend to change.
arXiv Detail & Related papers (2022-06-04T18:54:31Z) - Quasi-orthogonality and intrinsic dimensions as measures of learning and
generalisation [55.80128181112308]
We show that dimensionality and quasi-orthogonality of neural networks' feature space may jointly serve as network's performance discriminants.
Our findings suggest important relationships between the networks' final performance and properties of their randomly initialised feature spaces.
arXiv Detail & Related papers (2022-03-30T21:47:32Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Backprop-Free Reinforcement Learning with Active Neural Generative
Coding [84.11376568625353]
We propose a computational framework for learning action-driven generative models without backpropagation of errors (backprop) in dynamic environments.
We develop an intelligent agent that operates even with sparse rewards, drawing inspiration from the cognitive theory of planning as inference.
The robust performance of our agent offers promising evidence that a backprop-free approach for neural inference and learning can drive goal-directed behavior.
arXiv Detail & Related papers (2021-07-10T19:02:27Z) - Towards a Predictive Processing Implementation of the Common Model of
Cognition [79.63867412771461]
We describe an implementation of the common model of cognition grounded in neural generative coding and holographic associative memory.
The proposed system creates the groundwork for developing agents that learn continually from diverse tasks as well as model human performance at larger scales.
arXiv Detail & Related papers (2021-05-15T22:55:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.