Understanding Sample Efficiency in Predictive Coding

• URL: http://arxiv.org/abs/2605.11911v1
• Date: Tue, 12 May 2026 10:24:20 GMT
• Title: Understanding Sample Efficiency in Predictive Coding
• Abstract summary: Small scale experiments show that Predictive Coding (PC) enables learning that is more sample efficient and effective in many contexts.<n>We quantify the efficiency of learning in BP and PC through a metric called target alignment''<n>We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks.
• Score: 5.299779650604401
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predictive Coding (PC) is an influential account of cortical learning. Much of recent work has focused on comparing PC to Backpropagation (BP) to find whether PC offers any advantages. Small scale experiments show that PC enables learning that is more sample efficient and effective in many contexts, though a thorough theoretical understanding of the phenomena remains elusive. To address this, we quantify the efficiency of learning in BP and PC through a metric called ``target alignment'', which measures how closely the change in the output of the network is aligned to the output prediction error. We then derive and empirically validate analytical expressions for target alignment in Deep Linear Networks. We show that learning in PC is more efficient than BP, which is especially pronounced in deep, narrow and pre-trained networks. We also derive exact conditions for guaranteed optimal target alignment in PC and validate our findings through experiments. We study full training trajectories of linear and non-linear models, and find the predicted benefits of PC persist in practice even when some assumptions are violated. Overall, this work provides a mechanistic understanding of the higher learning efficiency observed for PC over BP in previous works, and can guide how PC should be parametrised to learn most effectively.

Related papers

• On the Infinite Width and Depth Limits of Predictive Coding Networks [8.779034498638826]
  Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP)<n>Recent work has improved the training stability of deep PC networks.<n>We study the infinite width and depth limits of PCNs.
  arXiv  Detail & Related papers  (2026-02-07T20:47:32Z)
• Efficient Online Learning with Predictive Coding Networks: Exploiting Temporal Correlations [26.073347035678342]
  Predictive Coding (PC) framework offers a biologically plausible alternative with local, Hebbian-like update rules.<n>We present Predictive Coding Network with Temporal Amortization (PCN-TA), which preserves latent states across temporal frames.<n>Experiments on the COIL-20 robotic perception dataset demonstrate that PCN-TA achieves 10% fewer weight updates compared to backpropagation.
  arXiv  Detail & Related papers  (2025-10-29T22:09:53Z)
• Towards Scaling Deep Neural Networks with Predictive Coding: Theory and Practice [1.2691047660244335]
  Backpropagation (BP) is the standard algorithm for training the deep neural networks that power modern artificial intelligence.<n>This thesis studies an alternative, potentially more efficient brain-inspired algorithm called predictive coding (PC)
  arXiv  Detail & Related papers  (2025-10-24T14:47:49Z)
• Tight Stability, Convergence, and Robustness Bounds for Predictive Coding Networks [60.3634789164648]
  Energy-based learning algorithms, such as predictive coding (PC), have garnered significant attention in the machine learning community. We rigorously analyze the stability, robustness, and convergence of PC through the lens of dynamical systems theory.
  arXiv  Detail & Related papers  (2024-10-07T02:57:26Z)
• Understanding Predictive Coding as an Adaptive Trust-Region Method [0.0]
  We develop a theory of PC as an adaptive trust-region (TR) algorithm that uses second-order information. We show that the learning dynamics of PC can be interpreted as interpolating between BP's loss gradient direction and a TR direction found by the PC inference dynamics.
  arXiv  Detail & Related papers  (2023-05-29T16:25:55Z)
• A Theoretical Framework for Inference and Learning in Predictive Coding Networks [41.58529335439799]
  Predictive coding (PC) is an influential theory in computational neuroscience. We provide a comprehensive theoretical analysis of the properties of PCNs trained with prospective configuration.
  arXiv  Detail & Related papers  (2022-07-21T04:17:55Z)
• Towards Scaling Difference Target Propagation by Learning Backprop Targets [64.90165892557776]
  Difference Target Propagation is a biologically-plausible learning algorithm with close relation with Gauss-Newton (GN) optimization. We propose a novel feedback weight training scheme that ensures both that DTP approximates BP and that layer-wise feedback weight training can be restored. We report the best performance ever achieved by DTP on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2022-01-31T18:20:43Z)
• A Theoretical View of Linear Backpropagation and Its Convergence [55.69505060636719]
  Backpropagation (BP) is widely used for calculating gradients in deep neural networks (DNNs) Recently, a linear variant of BP named LinBP was introduced for generating more transferable adversarial examples for performing black-box attacks. We provide theoretical analyses on LinBP in neural-network-involved learning tasks, including adversarial attack and model training.
  arXiv  Detail & Related papers  (2021-12-21T07:18:00Z)
• Efficient Estimation of Influence of a Training Instance [56.29080605123304]
  We propose an efficient method for estimating the influence of a training instance on a neural network model. Our method is inspired by dropout, which zero-masks a sub-network and prevents the sub-network from learning each training instance. We demonstrate that the proposed method can capture training influences, enhance the interpretability of error predictions, and cleanse the training dataset for improving generalization.
  arXiv  Detail & Related papers  (2020-12-08T04:31:38Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)
• Prototypical Contrastive Learning of Unsupervised Representations [171.3046900127166]
  Prototypical Contrastive Learning (PCL) is an unsupervised representation learning method. PCL implicitly encodes semantic structures of the data into the learned embedding space. PCL outperforms state-of-the-art instance-wise contrastive learning methods on multiple benchmarks.
  arXiv  Detail & Related papers  (2020-05-11T09:53:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.