Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study

• URL: http://arxiv.org/abs/2608.15483v1
• Date: Sun, 16 Aug 2026 02:17:51 GMT
• Title: Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Study
• Abstract summary: Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or redistributions.<n>We study short-horizon predictability as a measure of temporal redundancy, where, when, and under which training conditions recent updates contain information about near-future parameter motion.
• Score: 2.8019744694531554
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern deep networks are trained through long update trajectories, yet their temporal organization remains less systematically characterized than architectures, losses, or optimizers. We study short-horizon predictability as a measure of temporal redundancy: where, when, and under which training conditions recent updates contain information about near-future parameter motion. We combine three complementary probe families, displacement-direction, subspace-residual, and predictor-based probes, with convention-aware, null-calibrated group-level readouts, and apply them to multi-pass vision training on CIFAR and public Pythia pretraining checkpoints. Across both regimes, vector-like tensors such as normalization parameters and biases (auxiliary parameters) exhibit simpler short-horizon dynamics than matrix-like feature-transforming weights (bulk parameters), whose predictable behavior concentrates in localized, time-varying pockets. Agreement within and across probe families, and with independent trajectory diagnostics, indicates that these measurements capture intrinsic trajectory structure, while probe differences distinguish complementary forms of temporal organization. Controlled CIFAR comparisons further show that architecture and training recipe systematically modulate the measured structure. A Pythia-70M case study further exposes a sequence of role-, depth-, and scale-dependent events, including bulk ESA falling below the random sign-agreement level and the emergence and redistribution of predictable qkv pockets across layers. These results position short-horizon predictability as a retrospective, parameter-resolved diagnostic of training dynamics.

Related papers

• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• Learning Structural Convergence: A Neuro-Symbolic Benchmark for Temporal Reasoning [2.7644691581661287]
  This paper introduces TRACTA, a benchmark for temporal structural reasoning in high-complexity event-driven systems.<n>The benchmark includes three tasks: early_warning, pattern_detection, and run_classification.<n>Results show that raw event-level learning remains informative, but learned temporal modeling over semantic capability achieves the highest aggregate point estimates.
  arXiv  Detail & Related papers  (2026-07-24T14:50:19Z)
• Learning Spatio-Temporal Foundation Models from Pure Synthetic Data [85.23980441738811]
  NeoST is a scalable synthetic-training corpus to real-world bias, a latent-space reasoning architecture that generates and iteratively refines multiple future trajectories, and latent-space objectives that emphasize structural dynamics and enable inference-time correction under distribution shifts.<n>Extensive experiments across diverse real-world benchmarks show that NeoST consistently outperforms existing STFMs in diverse real-worldtemporal systems.
  arXiv  Detail & Related papers  (2026-06-27T02:56:46Z)
• Topological Out-of-Domain Generalization in Dynamical Systems Reconstruction [9.842968830321876]
  Predicting the behavior of dynamical systems (DS) beyond the dynamical and parameter regimes observed in training is a pivotal and essentially unresolved problem in scientific ML.<n>Recent hierarchical and hyper-network guided approaches for DS reconstruction (DSR) enable training on many DS simultaneously.<n>We propose a combination of remedies for these shortcomings, most importantly feature splitting, and furthermore derive a closed-form bound on the reliable extrapolation range.
  arXiv  Detail & Related papers  (2026-06-22T07:51:48Z)
• Navigating by Old Maps: The Pitfalls of Static Mechanistic Localization in LLM Post-Training [59.49057547517285]
  We systematically track the structural evolution of Transformer circuits throughout the supervised fine-tuning process.<n>Our empirical results reveal that circuits inherently exhibit "Free Evolution" during parameter updates.<n>This work underscores the necessity of "foresight" in mechanistic localization and proposes a framework for future research.
  arXiv  Detail & Related papers  (2026-05-07T11:59:52Z)
• Koopman Operator Identification of Model Parameter Trajectories for Temporal Domain Generalization (KOMET) [0.16311150636417257]
  We present KOMET, a model-agnostic, data-driven framework that treats the sequence of trained parameter vectors as the trajectory of a nonlinear dynamical system.<n>A warm-start sequential training protocol enforces parameter-trajectory smoothness, and a Fourier-augmented observable dictionary exploits the periodic structure inherent in many real-world distribution drifts.<n>KoMET achieves mean autonomous-rollout accuracies between 0.981 and 1.000 over 100 held-out time steps.
  arXiv  Detail & Related papers  (2026-03-27T18:58:45Z)
• VIKING: Deep variational inference with stochastic projections [48.946143517489496]
  Variational mean field approximations tend to struggle with contemporary overparametrized deep neural networks.<n>We propose a simple variational family that considers two independent linear subspaces of the parameter space.<n>This allows us to build a fully-correlated approximate posterior reflecting the overparametrization.
  arXiv  Detail & Related papers  (2025-10-27T15:38:35Z)
• Time-Scale Coupling Between States and Parameters in Recurrent Neural Networks [3.924071936547547]
  Gated neural networks (RNNs) implicitly induce adaptive learning-rate behavior.<n>Effect arises from the coupling between state-space time scales--parametrized by the gates--and parameter-space dynamics.<n> Empirical simulations corroborate these claims.
  arXiv  Detail & Related papers  (2025-08-16T18:19:34Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Stochastically forced ensemble dynamic mode decomposition for forecasting and analysis of near-periodic systems [65.44033635330604]
  We introduce a novel load forecasting method in which observed dynamics are modeled as a forced linear system. We show that its use of intrinsic linear dynamics offers a number of desirable properties in terms of interpretability and parsimony. Results are presented for a test case using load data from an electrical grid.
  arXiv  Detail & Related papers  (2020-10-08T20:25:52Z)
• Supporting Optimal Phase Space Reconstructions Using Neural Network Architecture for Time Series Modeling [68.8204255655161]
  We propose an artificial neural network with a mechanism to implicitly learn the phase spaces properties. Our approach is either as competitive as or better than most state-of-the-art strategies.
  arXiv  Detail & Related papers  (2020-06-19T21:04:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.