Error Highways: Scaling Predictive Coding to Very Deep Networks

• URL: http://arxiv.org/abs/2606.22744v1
• Date: Mon, 22 Jun 2026 01:12:29 GMT
• Title: Error Highways: Scaling Predictive Coding to Very Deep Networks
• Abstract summary: Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop)<n>A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from clamped boundaries.<n>We propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure.
• Score: 45.88028371034407
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Predictive coding networks (PCNs) offer a biologically-plausible, local-learning alternative to back-propagation of errors (backprop). Nevertheless, they have remained largely confined to shallow architectures and evaluated on simple machine intelligence benchmarks. A central obstacle to scaling PCNs is that the learning signal decays rapidly as it propagates away from the clamped boundaries, leaving interior layers effectively unchanged. To directly counter this problem, we propose highway error propagation (HEP), a scheme that augments the free energy function underlying predictive coding (PC) by altering its neural structure with feedback matrices $V_{L\to i}$ that couple selected hidden states directly to the clamped output error. Since this coupling is linear in the hidden state, the highway pathway delivers a correction at every inference step whose magnitude is independent of depth, in contrast to vanilla PC where the output error reaches the $i$-th hidden layer with attenuation that decays exponentially in depth. This bypasses the Jacobian chain while preserving the local PC synaptic update rule. On MNIST and Fashion-MNIST, we show that HEP effectively trains MLPs of up to 128 layers with accuracy that is robust with respect to depth.

Related papers

• Reducing ANN-SNN Conversion Error via Residual Membrane Potential Alignment [52.910770415431244]
  Spiking Neural Networks (SNNs) serve as core architectures for neuromorphic computing.<n>ANN-SNN conversion circumvents issues by reusing well-trained ANN weights for low-latency, energy-efficient inference.<n>Existing conversion schemes suffer from severe accuracy drops at small timesteps, large inference delays and cumulative quantization errors.
  arXiv  Detail & Related papers  (2026-08-14T04:46:03Z)
• Closed-form predictive coding via hierarchical Gaussian filters [33.92034506804344]
  Predictive coding offers a local and biologically grounded alternative to backpropagation in the training of artificial neural networks.<n>On FashionMNIST, our solution approaches backpropagation in epoch-level wall-clock cost while converging in fewer epochs.<n>We establish that closed-form variational inference with online precision learning provides a tractable foundation for deep predictive coding networks.
  arXiv  Detail & Related papers  (2026-05-19T10:11:41Z)
• A Theory of Generalization in Deep Learning [0.0]
  We present a non-asymptotic theory of generalization in deep learning where the empirical neural tangent kernel partitions the output space.<n>Within the signal channel, minibatch SGD ensures that coherent population signal accumulates via fast linear drift, while idiosyncratic memorization is suppressed into a slow, diffusive random walk.<n>This theory naturally explains disparate phenomena in deep learning theory, such as benign overfitting, double descent, implicit bias, and grokking.
  arXiv  Detail & Related papers  (2026-05-02T00:21:36Z)
• Accelerated Predictive Coding Networks via Direct Kolen-Pollack Feedback Alignment [7.328567184271344]
  Predictive coding (PC) is a biologically inspired algorithm for training neural networks that relies only on local updates.<n>We propose direct Kolen-Pollack predictive coding (DKP-PC)<n>It simultaneously addresses both feedback delay and exponential decay, yielding a more efficient and scalable variant of PC.
  arXiv  Detail & Related papers  (2026-02-17T13:29:14Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• Bio-Inspired Artificial Neural Networks based on Predictive Coding [7.500043032028289]
  Backpropagation (BP) of errors is the backbone training algorithm for artificial neural networks (ANNs)<n>BP uses the chain rule to propagate the loss gradient backward through the network hierarchy, allowing efficient weight updates.<n> Predictive Coding (PC) has gained interest as a biologically plausible alternative that updates weights using only local information.
  arXiv  Detail & Related papers  (2025-08-12T09:09:16Z)
• ePC: Overcoming Exponential Signal Decay in Deep Predictive Coding Networks [9.400040788307223]
  Predictive Coding (PC) offers a biologically plausible alternative to backpropagation for neural network training.<n>This paper identifies the root cause and provides a principled solution.
  arXiv  Detail & Related papers  (2025-05-26T15:39:16Z)
• Find A Winning Sign: Sign Is All We Need to Win the Lottery [52.63674911541416]
  We show that a sparse network trained by an existing IP method can retain its basin of attraction if its parameter signs and normalization layer parameters are preserved.<n>To take a step closer to finding a winning ticket, we alleviate the reliance on normalization layer parameters by preventing high error barriers along the linear path between the sparse network trained by our method and its counterpart with normalization layer parameters.
  arXiv  Detail & Related papers  (2025-04-07T09:30:38Z)
• LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers [0.0468732641979009]
  We propose a layer-wise parameterization for convolutional neural networks (CNNs) that includes built-in robustness guarantees. Our method Lip Kernel directly parameterizes dissipative convolution kernels using a 2-D Roesser-type state space model. We show that the run-time using our method is orders of magnitude faster than state-of-the-art Lipschitz-bounded networks.
  arXiv  Detail & Related papers  (2024-10-29T17:20:14Z)
• Towards Practical Control of Singular Values of Convolutional Layers [65.25070864775793]
  Convolutional neural networks (CNNs) are easy to train, but their essential properties, such as generalization error and adversarial robustness, are hard to control. Recent research demonstrated that singular values of convolutional layers significantly affect such elusive properties. We offer a principled approach to alleviating constraints of the prior art at the expense of an insignificant reduction in layer expressivity.
  arXiv  Detail & Related papers  (2022-11-24T19:09:44Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)
• LoCo: Local Contrastive Representation Learning [93.98029899866866]
  We show that by overlapping local blocks stacking on top of each other, we effectively increase the decoder depth and allow upper blocks to implicitly send feedbacks to lower blocks. This simple design closes the performance gap between local learning and end-to-end contrastive learning algorithms for the first time.
  arXiv  Detail & Related papers  (2020-08-04T05:41:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.