Hierarchical Predictive Coding Models in a Deep-Learning Framework
- URL: http://arxiv.org/abs/2005.03230v2
- Date: Wed, 23 Sep 2020 00:42:39 GMT
- Title: Hierarchical Predictive Coding Models in a Deep-Learning Framework
- Authors: Matin Hosseini, Anthony Maida
- Abstract summary: We review some of the more well known models of predictive coding.
We also survey some recent attempts to cast these models within a deep learning framework.
- Score: 1.370633147306388
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Bayesian predictive coding is a putative neuromorphic method for acquiring
higher-level neural representations to account for sensory input. Although
originating in the neuroscience community, there are also efforts in the
machine learning community to study these models. This paper reviews some of
the more well known models. Our review analyzes module connectivity and
patterns of information transfer, seeking to find general principles used
across the models. We also survey some recent attempts to cast these models
within a deep learning framework. A defining feature of Bayesian predictive
coding is that it uses top-down, reconstructive mechanisms to predict incoming
sensory inputs or their lower-level representations. Discrepancies between the
predicted and the actual inputs, known as prediction errors, then give rise to
future learning that refines and improves the predictive accuracy of learned
higher-level representations. Predictive coding models intended to describe
computations in the neocortex emerged prior to the development of deep learning
and used a communication structure between modules that we name the Rao-Ballard
protocol. This protocol was derived from a Bayesian generative model with some
rather strong statistical assumptions. The RB protocol provides a rubric to
assess the fidelity of deep learning models that claim to implement predictive
coding.
Related papers
- Evaluating and Explaining Large Language Models for Code Using Syntactic
Structures [74.93762031957883]
This paper introduces ASTxplainer, an explainability method specific to Large Language Models for code.
At its core, ASTxplainer provides an automated method for aligning token predictions with AST nodes.
We perform an empirical evaluation on 12 popular LLMs for code using a curated dataset of the most popular GitHub projects.
arXiv Detail & Related papers (2023-08-07T18:50:57Z) - Robust Graph Representation Learning via Predictive Coding [46.22695915912123]
Predictive coding is a message-passing framework initially developed to model information processing in the brain.
In this work, we build models that rely on the message-passing rule of predictive coding.
We show that the proposed models are comparable to standard ones in terms of performance in both inductive and transductive tasks.
arXiv Detail & Related papers (2022-12-09T03:58:22Z) - NCTV: Neural Clamping Toolkit and Visualization for Neural Network
Calibration [66.22668336495175]
A lack of consideration for neural network calibration will not gain trust from humans.
We introduce the Neural Clamping Toolkit, the first open-source framework designed to help developers employ state-of-the-art model-agnostic calibrated models.
arXiv Detail & Related papers (2022-11-29T15:03:05Z) - Pyramidal Predictive Network: A Model for Visual-frame Prediction Based
on Predictive Coding Theory [1.4610038284393165]
We propose a novel neural network model for the task of visual-frame prediction.
The model is composed of a series of recurrent and convolutional units forming the top-down and bottom-up streams.
It learns to predict future frames in a visual sequence, with ConvLSTMs on each layer in the network making local prediction from top to down.
arXiv Detail & Related papers (2022-08-15T06:28:34Z) - On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
Differentiable causal discovery has proposed to factorize the data generating process into a set of modules.
We study the generalization and adaption performance of such modular neural causal models.
Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
arXiv Detail & Related papers (2022-06-09T17:12:32Z) - Sparse Flows: Pruning Continuous-depth Models [107.98191032466544]
We show that pruning improves generalization for neural ODEs in generative modeling.
We also show that pruning finds minimal and efficient neural ODE representations with up to 98% less parameters compared to the original network, without loss of accuracy.
arXiv Detail & Related papers (2021-06-24T01:40:17Z) - Demystifying Code Summarization Models [5.608277537412537]
We evaluate four prominent code summarization models: extreme summarizer, code2vec, code2seq, and sequence GNN.
Results show that all models base their predictions on syntactic and lexical properties with little to none semantic implication.
We present a novel approach to explaining the predictions of code summarization models through the lens of training data.
arXiv Detail & Related papers (2021-02-09T03:17:46Z) - The Neural Coding Framework for Learning Generative Models [91.0357317238509]
We propose a novel neural generative model inspired by the theory of predictive processing in the brain.
In a similar way, artificial neurons in our generative model predict what neighboring neurons will do, and adjust their parameters based on how well the predictions matched reality.
arXiv Detail & Related papers (2020-12-07T01:20:38Z) - Relaxing the Constraints on Predictive Coding Models [62.997667081978825]
Predictive coding is an influential theory of cortical function which posits that the principal computation the brain performs is the minimization of prediction errors.
Standard implementations of the algorithm still involve potentially neurally implausible features such as identical forward and backward weights, backward nonlinear derivatives, and 1-1 error unit connectivity.
In this paper, we show that these features are not integral to the algorithm and can be removed either directly or through learning additional sets of parameters with Hebbian update rules without noticeable harm to learning performance.
arXiv Detail & Related papers (2020-10-02T15:21:37Z) - Introduction to Rare-Event Predictive Modeling for Inferential
Statisticians -- A Hands-On Application in the Prediction of Breakthrough
Patents [0.0]
We introduce a machine learning (ML) approach to quantitative analysis geared towards optimizing the predictive performance.
We discuss the potential synergies between the two fields against the backdrop of this, at first glance, target-incompatibility.
We are providing a hands-on predictive modeling introduction for a quantitative social science audience while aiming at demystifying computer science jargon.
arXiv Detail & Related papers (2020-03-30T13:06:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.