Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping

• URL: http://arxiv.org/abs/2608.01833v1
• Date: Mon, 03 Aug 2026 07:44:57 GMT
• Title: Tunneling the Loss Landscape: Bypassing Memorization with Monte Carlo Parameter Swapping
• Abstract summary: Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of memorization before abrupt generalization.<n>We show that standard optimization presents clear signatures of glass dynamics and traps the grokking network in a kinetic arrested state.<n>We introduce State-Aware Monte Carlo, gradient Swapping, an optimization plug-in that can accelerate generalization.
• Score: 10.558597505301611
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grokking is a striking phenomenon in neural network training, where a model can undergo a prolonged period of pure memorization before abrupt generalization. While previous works have attempted to interpret it through classical machine learning mechanisms like weight norm, recent research draws an analogy from statistical physics, framing grokking as a form of computational glass relaxation. This theory defines the initial memorization as a result of `fast cooling' where the training loss is reduced so quickly that a glass state is formed, followed by a `slow relaxation' towards final generalization. Although providing a unifying framework for representative grokking theories, this perspective has remained largely at the theoretical on macroscopic level without direct empirical validation on training dynamics. Here we introduce a three-component framework to directly characterize the training dynamics via parameter mobility (PM), and two representative measurements from glassy dynamics: replica correlation (RC) and fractal dimension (FD). We demonstrate that standard optimization presents clear signatures of glass dynamics and inherently traps the grokking network in a kinetic arrested memorization state with a collapsed mobility, strong history dependence, and channel-like motions. This quantitative agreement motivates us to introduce State-Aware Monte Carlo Parameter Swapping (SAM-Swap), an optimization plug-in that can accelerate generalization, inspired by swap Monte Carlo algorithm widely used in glass dynamics. Comparing SAM-Swap, weight decay, and Gaussian gradient noise, we find that accelerated generalization is consistently associated with random exploration in the parameter space, similar to diffusion in physics.

Related papers

• On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models [50.824624057327725]
  State Space Models (SSMs) have emerged as a compelling alternative to Transformers.<n>We show through theory and experiments that this gating mechanism causes SSMs to first learn an in-weights "memorization" solution.<n>On the other hand, we find that gating is often beneficial for improving generalization to long sequence lengths.
  arXiv  Detail & Related papers  (2026-09-15T02:36:13Z)
• Interpreting learning dynamics of autoencoders: Transient scaling and emerging concepts of the Ising model [0.0]
  unsupervised autoencoders trained on microscopic spin configurations from the Ising model learn macroscopic, theory-relevant variables.<n>We quantify learning across multiple spatial scales and reveal two distinct dynamical regimes.<n>A dynamical viewpoint of learning is established in which intrinsic properties expose the effects of forced changes in representation.
  arXiv  Detail & Related papers  (2026-07-11T12:31:34Z)
• OMNIFLOW: A Physics-Grounded Multimodal Agent for Generalized Scientific Reasoning [51.33849811496781]
  Large Language Models (LLMs) have demonstrated exceptional logical reasoning capabilities but frequently struggle with the continuous dynamics by Partial Differential Equations (PDEs)<n>We propose OMNIFLOW, a multimodal-symbolic architecture designed to ground frozen LLMs in fundamental physical laws without requiring domain-specific parameter updates.<n>We evaluate it on a benchmark spanning microscopic turbulence, theoretical Navier-Stokes, and macroscopic global weather forecasting.
  arXiv  Detail & Related papers  (2026-03-16T18:29:01Z)
• Digital Metabolism: Decoupling Logic from Facts via Regenerative Unlearning -- Towards a Pure Neural Logic Core [4.073707521515039]
  "Digital metabolism" is a hypothesis suggesting that targeted forgetting is necessary for distilling a pure neural logic core.<n>We introduce the Regenerative Logic-Core Protocol (RLCP), a dual-stream training framework that renders specific factual dependencies linearly undecodable.<n> Empirical analysis on GSM8K reveals that the "metabolized" model spontaneously adopts Symbolic chain-of-thought scaffolding.
  arXiv  Detail & Related papers  (2026-01-15T19:21:16Z)
• From Black Hole to Galaxy: Neural Operator: Framework for Accretion and Feedback Dynamics [70.27068115318681]
  We introduce a neural-based ''subgrid black hole'' that learns the small-scale local dynamics and embeds it within direct simulations.<n>Thanks to the great speedup in fine-scale evolution, our approach captures intrinsic variability in accretion-driven feedback, allowing dynamic coupling between the central black hole and galaxy-scale gas.
  arXiv  Detail & Related papers  (2025-12-01T11:47:49Z)
• The Geometry of Grokking: Norm Minimization on the Zero-Loss Manifold [5.076419064097734]
  We argue that post-memorization learning can be understood through the lens of constrained optimization.<n>We show that gradient descent effectively minimizes the weight norm on the zero-loss manifold.<n>Experiments confirm that simulating the training process using our predicted gradients reproduces both the delayed generalization and representation learning characteristic of grokking.
  arXiv  Detail & Related papers  (2025-11-02T18:44:42Z)
• Mimicking the Physicist's Eye:A VLM-centric Approach for Physics Formula Discovery [98.58830663687911]
  VIPERR-aq1 is a multimodal model that performs Visual Induction for Equation Reasoning.<n>It integrates visual perception, trajectory data, and symbolic reasoning to emulate the scientific discovery process.<n>It consistently outperforms state-of-the-art VLM baselines in accuracy and interpretability.
  arXiv  Detail & Related papers  (2025-08-24T14:34:21Z)
• Learning to Dissipate Energy in Oscillatory State-Space Models [51.98491034847041]
  State-space models (SSMs) are a class of networks for sequence learning.<n>We show that D-LinOSS consistently outperforms previous LinOSS methods on long-range learning tasks.
  arXiv  Detail & Related papers  (2025-05-17T23:15:17Z)
• Is Grokking a Computational Glass Relaxation? [11.327748614675885]
  We study the phenomenon of grokking, where neural networks abruptly generalize long after the training performance reaches a near-perfect level.<n>We find NO entropy barrier in the memorization-to-generalization transition of grokking, challenging previous theory that defines grokking as a first-order phase transition.<n>Inspired by grokking's far-from-equilibrium nature, we develop a toy WanD based on Wang-landau molecular dynamics, which can eliminate grokking without any constraints and find high-norm generalizing solutions.
  arXiv  Detail & Related papers  (2025-05-16T16:20:02Z)
• The Limiting Dynamics of SGD: Modified Loss, Phase Space Oscillations, and Anomalous Diffusion [29.489737359897312]
  We study the limiting dynamics of deep neural networks trained with gradient descent (SGD) We show that the key ingredient driving these dynamics is not the original training loss, but rather the combination of a modified loss, which implicitly regularizes the velocity and probability currents, which cause oscillations in phase space.
  arXiv  Detail & Related papers  (2021-07-19T20:18:57Z)
• Kernel and Rich Regimes in Overparametrized Models [69.40899443842443]
  We show that gradient descent on overparametrized multilayer networks can induce rich implicit biases that are not RKHS norms. We also demonstrate this transition empirically for more complex matrix factorization models and multilayer non-linear networks.
  arXiv  Detail & Related papers  (2020-02-20T15:43:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.