Speed Limit for Information Acquisition in Stochastic Learning Dynamics

• URL: http://arxiv.org/abs/2609.08219v1
• Date: Tue, 08 Sep 2026 04:02:52 GMT
• Title: Speed Limit for Information Acquisition in Stochastic Learning Dynamics
• Abstract summary: We derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information.<n>The resulting inequality decomposes the information flow into drift and noise contributions.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about latent variables in the data-generating process. The resulting inequality decomposes the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different latent variables are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for diagnosing when and how different aspects of the data-generating mechanism are acquired during stochastic learning.

Related papers

• Information theory and discriminative sampling for model discovery [2.47593085771929]
  We leverage the Fisher Information Matrix (FIM) within the data-driven framework of em identification of nonlinear dynamics (SINDy)<n>We visualize information patterns in chaotic and non-chaotic systems for both single trajectories and multiple initial conditions.
  arXiv  Detail & Related papers  (2025-12-17T22:08:21Z)
• Neural Ordinary Differential Equations for Learning and Extrapolating System Dynamics Across Bifurcations [1.4316136514132225]
  We show that Neural Ordinary Differential Equations can recover underlying bifurcation structures directly from time-series data.<n>We demonstrate our approach on three test cases: the Lorenz system transitioning from non-chaotic to chaotic behaviour, the R"ossler system moving from chaos to period doubling, and a predator-prey model exhibiting collapse via a global bifurcation.
  arXiv  Detail & Related papers  (2025-07-25T07:44:34Z)
• Localized Gaussians as Self-Attention Weights for Point Clouds Correspondence [92.07601770031236]
  We investigate semantically meaningful patterns in the attention heads of an encoder-only Transformer architecture.<n>We find that fixing the attention weights not only accelerates the training process but also enhances the stability of the optimization.
  arXiv  Detail & Related papers  (2024-09-20T07:41:47Z)
• Towards stable real-world equation discovery with assessing differentiating quality influence [52.2980614912553]
  We propose alternatives to the commonly used finite differences-based method. We evaluate these methods in terms of applicability to problems, similar to the real ones, and their ability to ensure the convergence of equation discovery algorithms.
  arXiv  Detail & Related papers  (2023-11-09T23:32:06Z)
• Ultimate limit on learning non-Markovian behavior: Fisher information rate and excess information [0.0]
  We address the fundamental limits of learning unknown parameters of any process from time-series data. We discover exact closed-form expressions for how optimal inference scales with observation length.
  arXiv  Detail & Related papers  (2023-10-06T01:53:42Z)
• Kalman Filter for Online Classification of Non-Stationary Data [101.26838049872651]
  In Online Continual Learning (OCL) a learning system receives a stream of data and sequentially performs prediction and training steps. We introduce a probabilistic Bayesian online learning model by using a neural representation and a state space model over the linear predictor weights. In experiments in multi-class classification we demonstrate the predictive ability of the model and its flexibility to capture non-stationarity.
  arXiv  Detail & Related papers  (2023-06-14T11:41:42Z)
• Tracking changes using Kullback-Leibler divergence for the continual learning [2.0305676256390934]
  This article introduces a novel method for monitoring changes in the probabilistic distribution of multi-dimensional data streams. As a measure of the rapidity of changes, we analyze the popular Kullback-Leibler divergence. We show how to use this metric to predict the concept drift occurrence and understand its nature.
  arXiv  Detail & Related papers  (2022-10-10T17:30:41Z)
• A Causality-Based Learning Approach for Discovering the Underlying Dynamics of Complex Systems from Partial Observations with Stochastic Parameterization [1.2882319878552302]
  This paper develops a new iterative learning algorithm for complex turbulent systems with partial observations. It alternates between identifying model structures, recovering unobserved variables, and estimating parameters. Numerical experiments show that the new algorithm succeeds in identifying the model structure and providing suitable parameterizations for many complex nonlinear systems.
  arXiv  Detail & Related papers  (2022-08-19T00:35:03Z)
• Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
  We propose a novel approach to identify meaningful and independent factors of variation in a dataset. Our method involves two separate latent subspaces for the target property and the remaining input information. We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
  arXiv  Detail & Related papers  (2021-11-25T17:33:12Z)
• Encoding Domain Information with Sparse Priors for Inferring Explainable Latent Variables [2.8935588665357077]
  We propose spex-LVM, a factorial latent variable model with sparse priors to encourage the inference of explainable factors. spex-LVM utilizes existing knowledge of curated biomedical pathways to automatically assign annotated attributes to latent factors. Evaluations on simulated and real single-cell RNA-seq datasets demonstrate that our model robustly identifies relevant structure in an inherently explainable manner.
  arXiv  Detail & Related papers  (2021-07-08T10:19:32Z)
• Meta-learning using privileged information for dynamics [66.32254395574994]
  We extend the Neural ODE Process model to use additional information within the Learning Using Privileged Information setting. We validate our extension with experiments showing improved accuracy and calibration on simulated dynamics tasks.
  arXiv  Detail & Related papers  (2021-04-29T12:18:02Z)
• Leveraging Global Parameters for Flow-based Neural Posterior Estimation [90.21090932619695]
  Inferring the parameters of a model based on experimental observations is central to the scientific method. A particularly challenging setting is when the model is strongly indeterminate, i.e., when distinct sets of parameters yield identical observations. We present a method for cracking such indeterminacy by exploiting additional information conveyed by an auxiliary set of observations sharing global parameters.
  arXiv  Detail & Related papers  (2021-02-12T12:23:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.