A Function-Space Approach to the Statistical Mechanics of Learning Dynamics

• URL: http://arxiv.org/abs/2609.09589v1
• Date: Wed, 09 Sep 2026 01:30:14 GMT
• Title: A Function-Space Approach to the Statistical Mechanics of Learning Dynamics
• Abstract summary: We develop a statistical-mechanical description of learning directly in function space.<n>For mean-squared loss, the exact error dynamics are governed by the learning operator (M=JJast)<n>For ReLU-type function spaces under mild stable statistical conditions, (B=_2Lastmathcal K L) measures coarse-grained second-order structure.
• Score: 16.495935745587936
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks exhibit regular macroscopic behavior despite highly nonlinear dynamics in vast parameter spaces. We develop a statistical-mechanical description of learning directly in function space, treating parameter configurations as microscopic realizations and functions with their dynamical operators as macroscopic variables. For mean-squared loss, the exact error dynamics are governed by the learning operator \(M=JJ^\ast\). Combining the dynamical Boltzmann weight of the conditional stochastic dynamics with the parameter-space density of states, whose local curvature defines a statistical operator \(B\), and integrating over local fluctuations yields $$ Φ_{\mathrm{fluc}}(M;B)=\frac{σ_ξ^2}{2}\log\det(M^{-1}+B)+\mathrm{const}. $$ At fixed spectrum, this term is rotationally stationary when \([M,B]=0\), is minimized by pairing large eigenvalues of \(M\) with small eigenvalues of \(B\), and generates a local restoring contribution against rotational mismatch. For ReLU-type function spaces under mild stable statistical conditions, \(B=σ_ξ^2L^\ast\mathcal K L\), where \(L\) measures coarse-grained second-order structure. Thus the low-\(B\) sector corresponds, up to bounded anisotropy of \(\mathcal K\), to low structural curvature, implying a preference for faster relaxation along smooth, data-adaptive directions. These results identify function space as a natural macroscopic level for studying stable collective organization in learning.

Related papers

• Scalable Lindblad Noise Learning via Stochastic Tensor-Network Simulation [48.887627688666974]
  Learning dissipation rates in large-scale open quantum systems is a major obstacle for near-term quantum technologies.<n>Here, we propose a scalable noise-learning framework for Lindblad dissipation rates that combines a simulation method, the Jump Method (TJM)<n>We demonstrate the approach on two noise models in the Ising model, in which independent dissipation rates are learned for each site up to $N_mathrmsite=16$.<n>This combination of scalable numerics and rigorous theoretical guarantees positions TJM-based noise learning as a practical foundation for characterizing dissipation in large quantum devices.
  arXiv  Detail & Related papers  (2026-08-25T15:07:11Z)
• Effective Covariance Dynamics in Solvable High-Dimensional GANs [2.639628765667673]
  We study a solvable high-dimensional model of generative adversarial network (GAN) training in which a linear generator learns a low-dimensional subspace from data.<n>We prove that the microscopic training process converges, in the high-dimensional limit, to deterministic ordinary differential equations governed by this effective covariance.
  arXiv  Detail & Related papers  (2026-06-25T16:33:54Z)
• Anderson Localization: A Floquet operator Krylov space perspective [0.0]
  We show that studying the dynamics at stroboscopic rather than continuous times has its advantages.<n>For stroboscopic dynamics, the operator Krylov space corresponds to the dynamics of an edge operator of an inhomogeneous Floquet transverse field Ising model.<n>The localization-delocalization transition is also demonstrated in operator Krylov space.
  arXiv  Detail & Related papers  (2026-05-22T18:22:47Z)
• Expectation-Maximization as a Spectrally Governed Relaxation Flow [10.371607616433966]
  We make explicit the latent-variable operator that connects global descent and local convergence.<n>Our results place global descent, local spectral behavior, and optimal local relaxation within a common dynamical framework.
  arXiv  Detail & Related papers  (2026-05-08T14:49:24Z)
• The Curvature Rate λ: A Scalar Measure of Input-Space Sharpness in Neural Networks [0.0]
  Curvature influences generalization, robustness, and how reliably neural networks respond to small input perturbations.<n>We introduce a scalar curvature measure defined directly in input space: the curvature rate lambda.<n>lambda tracks the emergence of high-frequency structure in the decision boundary.
  arXiv  Detail & Related papers  (2025-11-03T10:46:03Z)
• Nonparametric learning of stochastic differential equations from sparse and noisy data [2.389598109913754]
  We learn the entire drift function directly from data without strong structural assumptions.<n>We develop an Expectation-Maximization (EM) algorithm that employs a novel Sequential Monte Carlo (SMC) method.<n>The resulting EM-SMC-RKHS procedure enables accurate estimation of the drift function of dynamical systems in low-data regimes.
  arXiv  Detail & Related papers  (2025-08-15T17:01:59Z)
• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• Kernel-based off-policy estimation without overlap: Instance optimality beyond semiparametric efficiency [53.90687548731265]
  We study optimal procedures for estimating a linear functional based on observational data. For any convex and symmetric function class $mathcalF$, we derive a non-asymptotic local minimax bound on the mean-squared error.
  arXiv  Detail & Related papers  (2023-01-16T02:57:37Z)
• Implicit Bias of MSE Gradient Optimization in Underparameterized Neural Networks [0.0]
  We study the dynamics of a neural network in function space when optimizing the mean squared error via gradient flow. We show that the network learns eigenfunctions of an integral operator $T_Kinfty$ determined by the Neural Tangent Kernel (NTK) We conclude that damped deviations offers a simple and unifying perspective of the dynamics when optimizing the squared error.
  arXiv  Detail & Related papers  (2022-01-12T23:28:41Z)
• Sinkhorn Natural Gradient for Generative Models [125.89871274202439]
  We propose a novel Sinkhorn Natural Gradient (SiNG) algorithm which acts as a steepest descent method on the probability space endowed with the Sinkhorn divergence. We show that the Sinkhorn information matrix (SIM), a key component of SiNG, has an explicit expression and can be evaluated accurately in complexity that scales logarithmically. In our experiments, we quantitatively compare SiNG with state-of-the-art SGD-type solvers on generative tasks to demonstrate its efficiency and efficacy of our method.
  arXiv  Detail & Related papers  (2020-11-09T02:51:17Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.