Low Tensor Rank Learning of Neural Dynamics
- URL: http://arxiv.org/abs/2308.11567v2
- Date: Sat, 4 Nov 2023 11:47:43 GMT
- Title: Low Tensor Rank Learning of Neural Dynamics
- Authors: Arthur Pellegrino, N Alex Cayco-Gajic, Angus Chadwick
- Abstract summary: We show that low-tensor-rank weights emerge naturally in RNNs trained to solve low-dimensional tasks.
Our findings provide insight on the evolution of population connectivity over learning in both biological and artificial neural networks.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Learning relies on coordinated synaptic changes in recurrently connected
populations of neurons. Therefore, understanding the collective evolution of
synaptic connectivity over learning is a key challenge in neuroscience and
machine learning. In particular, recent work has shown that the weight matrices
of task-trained RNNs are typically low rank, but how this low rank structure
unfolds over learning is unknown. To address this, we investigate the rank of
the 3-tensor formed by the weight matrices throughout learning. By fitting RNNs
of varying rank to large-scale neural recordings during a motor learning task,
we find that the inferred weights are low-tensor-rank and therefore evolve over
a fixed low-dimensional subspace throughout the entire course of learning. We
next validate the observation of low-tensor-rank learning on an RNN trained to
solve the same task. Finally, we present a set of mathematical results bounding
the matrix and tensor ranks of gradient descent learning dynamics which show
that low-tensor-rank weights emerge naturally in RNNs trained to solve
low-dimensional tasks. Taken together, our findings provide insight on the
evolution of population connectivity over learning in both biological and
artificial neural networks, and enable reverse engineering of learning-induced
changes in recurrent dynamics from large-scale neural recordings.
Related papers
- Hebbian Learning based Orthogonal Projection for Continual Learning of
Spiking Neural Networks [74.3099028063756]
We develop a new method with neuronal operations based on lateral connections and Hebbian learning.
We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities.
Our method consistently solves for spiking neural networks with nearly zero forgetting.
arXiv Detail & Related papers (2024-02-19T09:29:37Z) - Spiking mode-based neural networks [2.5690340428649328]
Spiking neural networks play an important role in brain-like neuromorphic computations and in studying working mechanisms of neural circuits.
One drawback of training a large scale spiking neural network is that updating all weights is quite expensive.
We propose a spiking mode-based training protocol, where the recurrent weight matrix is explained as a Hopfield-like multiplication of three matrices.
arXiv Detail & Related papers (2023-10-23T06:54:17Z) - Interpretable statistical representations of neural population dynamics and geometry [4.459704414303749]
We introduce a representation learning method, MARBLE, that decomposes on-manifold dynamics into local flow fields and maps them into a common latent space.
In simulated non-linear dynamical systems, recurrent neural networks, and experimental single-neuron recordings from primates and rodents, we discover emergent low-dimensional latent representations.
These representations are consistent across neural networks and animals, enabling the robust comparison of cognitive computations.
arXiv Detail & Related papers (2023-04-06T21:11:04Z) - Heterogeneous Recurrent Spiking Neural Network for Spatio-Temporal
Classification [13.521272923545409]
Spi Neural Networks are often touted as brain-inspired learning models for the third wave of Artificial Intelligence.
This paper presents a heterogeneous spiking neural network (HRSNN) with unsupervised learning for video recognition tasks.
We show that HRSNN can achieve similar performance to state-of-the-temporal backpropagation trained supervised SNN, but with less computation.
arXiv Detail & Related papers (2022-09-22T16:34:01Z) - When, where, and how to add new neurons to ANNs [3.0969191504482243]
Neurogenesis in ANNs is an understudied and difficult problem, even compared to other forms of structural learning like pruning.
We introduce a framework for studying the various facets of neurogenesis: when, where, and how to add neurons during the learning process.
arXiv Detail & Related papers (2022-02-17T09:32:08Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - Dynamic Neural Diversification: Path to Computationally Sustainable
Neural Networks [68.8204255655161]
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks.
We explore the diversity of the neurons within the hidden layer during the learning process.
We analyze how the diversity of the neurons affects predictions of the model.
arXiv Detail & Related papers (2021-09-20T15:12:16Z) - Artificial Neural Variability for Deep Learning: On Overfitting, Noise
Memorization, and Catastrophic Forgetting [135.0863818867184]
artificial neural variability (ANV) helps artificial neural networks learn some advantages from natural'' neural networks.
ANV plays as an implicit regularizer of the mutual information between the training data and the learned model.
It can effectively relieve overfitting, label noise memorization, and catastrophic forgetting at negligible costs.
arXiv Detail & Related papers (2020-11-12T06:06:33Z) - Generalizable Machine Learning in Neuroscience using Graph Neural
Networks [0.0]
We show that neural networks perform remarkably well on both neuron-level dynamics prediction, and behavioral state classification.
In our experiments, we found that graph neural networks generally outperformed structure models and excel in generalization on unseen organisms.
arXiv Detail & Related papers (2020-10-16T18:09:46Z) - Learning compositional functions via multiplicative weight updates [97.9457834009578]
We show that multiplicative weight updates satisfy a descent lemma tailored to compositional functions.
We show that Madam can train state of the art neural network architectures without learning rate tuning.
arXiv Detail & Related papers (2020-06-25T17:05:19Z) - Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy.
We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
arXiv Detail & Related papers (2020-02-02T21:09:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.