A Study of Hidden-State Optimization Order in Predictive Coding Networks

• URL: http://arxiv.org/abs/2609.00686v1
• Date: Tue, 01 Sep 2026 04:07:04 GMT
• Title: A Study of Hidden-State Optimization Order in Predictive Coding Networks
• Abstract summary: Local learning methods offer an alternative to end-to-end backpropagation, but their unstructured local objectives can produce weak feature learning in deep networks.<n>We propose a boundary-first inference schedule that partitions a model into chunks, first coordinates hidden states at chunk boundaries, and then refines representations within each chunk.<n>We instantiate this schedule in predictive coding networks (PCNs), a local-learning framework in which hidden activities and prediction errors are explicitly exposed during inference.
• Score: 5.721855749950785
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Local learning methods offer an alternative to end-to-end backpropagation, but their unstructured local objectives can produce weak feature learning in deep networks. We study whether the order of hidden-state optimization can address this limitation. We propose a boundary-first inference schedule that partitions a model into chunks, first coordinates hidden states at chunk boundaries, and then refines representations within each chunk. We instantiate this schedule in predictive coding networks (PCNs), a local-learning framework in which hidden activities and prediction errors are explicitly exposed during inference. On CIFAR-10, the resulting boundary-first predictive-coding instantiation improves accuracy over standard predictive coding by $9.77\%$ under a standard parametrization and by $5.51\%$ under a $μ$-parametrization. Diagnostic analyses further show more non-trivial early-layer updates, lower initial-to-final CKA, and more diverse layerwise gradients, consistent with stronger feature learning. These results support boundary-first, chunk-based inference as a practical design principle for predictive-coding training and motivate its study in broader local-learning systems.

Related papers

• From Regression to Inference: Meta-Learning Predictors for Neural Architecture Search [9.003566823725441]
  Prediction-based approaches are widely used in neural architecture search (NAS), where a predictor estimates the performance of candidate architectures to guide selection.<n>Existing predictors are typically trained via supervised regression on limited samples, leading to overfitting and poor generalization to unseen architectures.<n>We propose a fundamentally different formulation that models performance prediction as a conditional function inference problem using a Convolutional Neural Process (ConvNP) with meta-learning capabilities.
  arXiv  Detail & Related papers  (2026-05-10T03:34:39Z)
• On Neural Scaling Laws for Weather Emulation through Continual Training [76.70310203875866]
  We study neural scaling in Scientific Machine Learning, focusing on models for weather forecasting.<n>To analyze scaling behavior in as simple a setting as possible, we adopt a minimal, scalable, general-purpose Swin Transformer architecture.<n>We show that models trained in this minimalist way follow predictable scaling trends and even outperform standard cosine learning rate schedules.
  arXiv  Detail & Related papers  (2026-03-26T17:37:25Z)
• Faster Predictive Coding Networks via Better Initialization [52.419343840654186]
  We propose a new technique for predictive coding networks that aims to preserve the iterative progress made on previous training samples.<n>Our experiments demonstrate substantial improvements in convergence speed and final test loss in both supervised and unsupervised settings.
  arXiv  Detail & Related papers  (2026-01-28T08:52:19Z)
• Multigrade Neural Network Approximation [13.496991650323038]
  We develop a principled framework for structured error refinement in deep neural networks.<n>We show that for any continuous target function, there exists a fixed-width block multigrade $textttRe$LU whose residuals decrease across grades and converge uniformly.
  arXiv  Detail & Related papers  (2026-01-23T16:46:25Z)
• Deep Hierarchical Learning with Nested Subspace Networks [53.71337604556311]
  We propose Nested Subspace Networks (NSNs) for large neural networks.<n>NSNs enable a single model to be dynamically and granularly adjusted across a continuous spectrum of compute budgets.<n>We show that NSNs can be surgically applied to pre-trained LLMs and unlock a smooth and predictable compute-performance frontier.
  arXiv  Detail & Related papers  (2025-09-22T15:13:14Z)
• BinConv: A Neural Architecture for Ordinal Encoding in Time-Series Forecasting [5.827431686047649]
  We propose textbfBinConv, a fully convolutional neural network architecture designed for probabilistic forecasting.<n>BinConv achieves superior performance compared to widely used baseline datasets in both point and probabilistic forecasting.
  arXiv  Detail & Related papers  (2025-05-30T13:41:39Z)
• Online Decision-Focused Learning [74.3205104323777]
  Decision-focused learning (DFL) is an increasingly popular paradigm for training models whose predictive outputs are used in decision-making tasks.<n>In this paper, we regularize the objective function to make it different and investigate how to overcome nonoptimality function.<n>We also showcase the effectiveness of our algorithms on a knapsack experiment, where they outperform two standard benchmarks.
  arXiv  Detail & Related papers  (2025-05-19T10:40:30Z)
• Improving Continual Learning Performance and Efficiency with Auxiliary Classifiers [12.561018201238152]
  We show that intermediate representations in neural network layers are less prone to forgetting, highlighting their potential to accelerate computation.<n>We also leverage the ACs to reduce the average cost of the inference by 10-60% without compromising accuracy.
  arXiv  Detail & Related papers  (2024-03-12T08:33:26Z)
• Adaptive Training Distributions with Scalable Online Bilevel Optimization [26.029033134519604]
  Large neural networks pretrained on web-scale corpora are central to modern machine learning. This work considers modifying the pretraining distribution in the case where one has a small sample of data reflecting the targeted test conditions. We propose an algorithm motivated by a recent formulation of this setting as an online, bilevel optimization problem.
  arXiv  Detail & Related papers  (2023-11-20T18:01:29Z)
• The Cascaded Forward Algorithm for Neural Network Training [61.06444586991505]
  We propose a new learning framework for neural networks, namely Cascaded Forward (CaFo) algorithm, which does not rely on BP optimization as that in FF. Unlike FF, our framework directly outputs label distributions at each cascaded block, which does not require generation of additional negative samples. In our framework each block can be trained independently, so it can be easily deployed into parallel acceleration systems.
  arXiv  Detail & Related papers  (2023-03-17T02:01:11Z)
• Learning Large-scale Neural Fields via Context Pruned Meta-Learning [60.93679437452872]
  We introduce an efficient optimization-based meta-learning technique for large-scale neural field training. We show how gradient re-scaling at meta-test time allows the learning of extremely high-quality neural fields. Our framework is model-agnostic, intuitive, straightforward to implement, and shows significant reconstruction improvements for a wide range of signals.
  arXiv  Detail & Related papers  (2023-02-01T17:32:16Z)
• Robust Learning of Parsimonious Deep Neural Networks [0.0]
  We propose a simultaneous learning and pruning algorithm capable of identifying and eliminating irrelevant structures in a neural network. We derive a novel hyper-prior distribution over the prior parameters that is crucial for their optimal selection. We evaluate the proposed algorithm on the MNIST data set and commonly used fully connected and convolutional LeNet architectures.
  arXiv  Detail & Related papers  (2022-05-10T03:38:55Z)
• Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization [52.7137956951533]
  We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
  arXiv  Detail & Related papers  (2022-02-14T16:42:16Z)
• Deep learning: a statistical viewpoint [120.94133818355645]
  Deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-perfect solutions to non-optimal training problems. We conjecture that specific principles underlie these phenomena.
  arXiv  Detail & Related papers  (2021-03-16T16:26:36Z)
• Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization [16.85167651136133]
  We take a broader view of training sparse networks and consider the role of regularization, optimization and architecture choices on sparse models. We show that gradient flow in sparse networks can be improved by reconsidering aspects of the architecture design and the training regime.
  arXiv  Detail & Related papers  (2021-02-02T18:40:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.