A Spectral Theory of Neural Prediction and Alignment
- URL: http://arxiv.org/abs/2309.12821v2
- Date: Mon, 11 Dec 2023 20:00:10 GMT
- Title: A Spectral Theory of Neural Prediction and Alignment
- Authors: Abdulkadir Canatar, Jenelle Feather, Albert Wakhloo, SueYeon Chung
- Abstract summary: We use a recent theoretical framework that relates the generalization error from regression to the spectral properties of the model and the target.
We test a large number of deep neural networks that predict visual cortical activity and show that there are multiple types of geometries that result in low neural prediction error as measured via regression.
- Score: 8.65717258105897
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The representations of neural networks are often compared to those of
biological systems by performing regression between the neural network
responses and those measured from biological systems. Many different
state-of-the-art deep neural networks yield similar neural predictions, but it
remains unclear how to differentiate among models that perform equally well at
predicting neural responses. To gain insight into this, we use a recent
theoretical framework that relates the generalization error from regression to
the spectral properties of the model and the target. We apply this theory to
the case of regression between model activations and neural responses and
decompose the neural prediction error in terms of the model eigenspectra,
alignment of model eigenvectors and neural responses, and the training set
size. Using this decomposition, we introduce geometrical measures to interpret
the neural prediction error. We test a large number of deep neural networks
that predict visual cortical activity and show that there are multiple types of
geometries that result in low neural prediction error as measured via
regression. The work demonstrates that carefully decomposing representational
metrics can provide interpretability of how models are capturing neural
activity and points the way towards improved models of neural activity.
Related papers
- Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
We propose to represent neural networks as computational graphs of parameters.
Our approach enables a single model to encode neural computational graphs with diverse architectures.
We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
arXiv Detail & Related papers (2024-03-18T18:01:01Z) - A Survey on Statistical Theory of Deep Learning: Approximation, Training Dynamics, and Generative Models [13.283281356356161]
We review the literature on statistical theories of neural networks from three perspectives.
Results on excess risks for neural networks are reviewed.
Papers that attempt to answer how the neural network finds the solution that can generalize well on unseen data'' are reviewed.
arXiv Detail & Related papers (2024-01-14T02:30:19Z) - Instance-wise Linearization of Neural Network for Model Interpretation [13.583425552511704]
The challenge can dive into the non-linear behavior of the neural network.
For a neural network model, the non-linear behavior is often caused by non-linear activation units of a model.
We propose an instance-wise linearization approach to reformulates the forward computation process of a neural network prediction.
arXiv Detail & Related papers (2023-10-25T02:07:39Z) - Spiking neural network for nonlinear regression [68.8204255655161]
Spiking neural networks carry the potential for a massive reduction in memory and energy consumption.
They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware.
A framework for regression using spiking neural networks is proposed.
arXiv Detail & Related papers (2022-10-06T13:04:45Z) - Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs.
By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
arXiv Detail & Related papers (2022-02-01T17:11:13Z) - On the relationship between predictive coding and backpropagation [0.0]
Predictive coding has been proposed as a potentially more biologically realistic alternative to backpropagation for training neural networks.
This manuscript reviews and extends recent work on the mathematical relationship between predictive coding and backpropagation for training feedforward artificial neural networks on supervised learning tasks.
arXiv Detail & Related papers (2021-06-20T18:22:50Z) - Persistent Homology Captures the Generalization of Neural Networks
Without A Validation Set [0.0]
We suggest studying the training of neural networks with Algebraic Topology, specifically Persistent Homology.
Using simplicial complex representations of neural networks, we study the PH diagram distance evolution on the neural network learning process.
Results show that the PH diagram distance between consecutive neural network states correlates with the validation accuracy.
arXiv Detail & Related papers (2021-05-31T09:17:31Z) - The Neural Coding Framework for Learning Generative Models [91.0357317238509]
We propose a novel neural generative model inspired by the theory of predictive processing in the brain.
In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
arXiv Detail & Related papers (2020-12-07T01:20:38Z) - A Bayesian Perspective on Training Speed and Model Selection [51.15664724311443]
We show that a measure of a model's training speed can be used to estimate its marginal likelihood.
We verify our results in model selection tasks for linear models and for the infinite-width limit of deep neural networks.
Our results suggest a promising new direction towards explaining why neural networks trained with gradient descent are biased towards functions that generalize well.
arXiv Detail & Related papers (2020-10-27T17:56:14Z) - Neural Networks with Recurrent Generative Feedback [61.90658210112138]
We instantiate this design on convolutional neural networks (CNNs)
In the experiments, CNN-F shows considerably improved adversarial robustness over conventional feedforward CNNs on standard benchmarks.
arXiv Detail & Related papers (2020-07-17T19:32:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.