A Gaussian Comparison Theorem for Training Dynamics in Machine Learning

• URL: http://arxiv.org/abs/2603.09310v1
• Date: Tue, 10 Mar 2026 07:42:36 GMT
• Title: A Gaussian Comparison Theorem for Training Dynamics in Machine Learning
• Abstract summary: We study training algorithms with data following a Gaussian mixture model.<n>We present a non-asymptotic result, connecting the evolution of the model to a surrogate dynamical system.
• Score: 4.277729131346358
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study training algorithms with data following a Gaussian mixture model. For a specific family of such algorithms, we present a non-asymptotic result, connecting the evolution of the model to a surrogate dynamical system, which can be easier to analyze. The proof of our result is based on the celebrated Gordon comparison theorem. Using our theorem, we rigorously prove the validity of the dynamic mean-field (DMF) expressions in the asymptotic scenarios. Moreover, we suggest an iterative refinement scheme to obtain more accurate expressions in non-asymptotic scenarios. We specialize our theory to the analysis of training a perceptron model with a generic first-order (full-batch) algorithm and demonstrate that fluctuation parameters in a non-asymptotic domain emerge in addition to the DMF kernels.

Related papers

• Rigorous Asymptotics for First-Order Algorithms Through the Dynamical Cavity Method [14.739377092233738]
  We formalize the dynamical cavity method and use it to give a new proof of the DMFT equations for General First Order Methods.<n>DMFT provides an description of the dynamics of macroscopic observables in certain disordered systems.
  arXiv  Detail & Related papers  (2026-03-15T19:51:41Z)
• InfoNCE Induces Gaussian Distribution [7.8922077372145685]
  A loss in contrastive training is InfoNCE and its variants.<n>We show that the InfoNCE objective induces Gaussian structure in representations that emerge from contrastive training.<n>The resulting Gaussian model enables principled analytical treatment of learned representations and is expected to support a wide range of applications in contrastive learning.
  arXiv  Detail & Related papers  (2026-02-27T13:35:58Z)
• Self-Supervised Coarsening of Unstructured Grid with Automatic Differentiation [55.88862563823878]
  In this work, we present an original algorithm to coarsen an unstructured grid based on the concepts of differentiable physics.<n>We demonstrate performance of the algorithm on two PDEs: a linear equation which governs slightly compressible fluid flow in porous media and the wave equation.<n>Our results show that in the considered scenarios, we reduced the number of grid points up to 10 times while preserving the modeled variable dynamics in the points of interest.
  arXiv  Detail & Related papers  (2025-07-24T11:02:13Z)
• Stochastic Gradient Descent for Gaussian Processes Done Right [86.83678041846971]
  We show that when emphdone right -- by which we mean using specific insights from optimisation and kernel communities -- gradient descent is highly effective. We introduce a emphstochastic dual descent algorithm, explain its design in an intuitive manner and illustrate the design choices. Our method places Gaussian process regression on par with state-of-the-art graph neural networks for molecular binding affinity prediction.
  arXiv  Detail & Related papers  (2023-10-31T16:15:13Z)
• A Geometric Perspective on Diffusion Models [57.27857591493788]
  We inspect the ODE-based sampling of a popular variance-exploding SDE. We establish a theoretical relationship between the optimal ODE-based sampling and the classic mean-shift (mode-seeking) algorithm.
  arXiv  Detail & Related papers  (2023-05-31T15:33:16Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• Distributed Bayesian Learning of Dynamic States [65.7870637855531]
  The proposed algorithm is a distributed Bayesian filtering task for finite-state hidden Markov models. It can be used for sequential state estimation, as well as for modeling opinion formation over social networks under dynamic environments.
  arXiv  Detail & Related papers  (2022-12-05T19:40:17Z)
• Double Descent in Random Feature Models: Precise Asymptotic Analysis for General Convex Regularization [4.8900735721275055]
  We provide precise expressions for the generalization of regression under a broad class of convex regularization terms. We numerically demonstrate the predictive capacity of our framework, and show experimentally that the predicted test error is accurate even in the non-asymptotic regime.
  arXiv  Detail & Related papers  (2022-04-06T08:59:38Z)
• Fractal Structure and Generalization Properties of Stochastic Optimization Algorithms [71.62575565990502]
  We prove that the generalization error of an optimization algorithm can be bounded on the complexity' of the fractal structure that underlies its generalization measure. We further specialize our results to specific problems (e.g., linear/logistic regression, one hidden/layered neural networks) and algorithms.
  arXiv  Detail & Related papers  (2021-06-09T08:05:36Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)
• Understanding the dynamics of message passing algorithms: a free probability heuristics [2.8021833233819486]
  We analyze the behavior of inference algorithms for probabilistic models with dense coupling matrices in the limit of large systems. For a toy Ising model, we are able to recover previous results such as the property of vanishing effective memories and the analytical convergence rate of the algorithm.
  arXiv  Detail & Related papers  (2020-02-03T19:50:31Z)
• Analysis of Bayesian Inference Algorithms by the Dynamical Functional Approach [2.8021833233819486]
  We analyze an algorithm for approximate inference with large Gaussian latent variable models in a student-trivial scenario. For the case of perfect data-model matching, the knowledge of static order parameters derived from the replica method allows us to obtain efficient algorithmic updates.
  arXiv  Detail & Related papers  (2020-01-14T17:22:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.