Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?

• URL: http://arxiv.org/abs/2608.14803v1
• Date: Fri, 14 Aug 2026 18:16:47 GMT
• Title: Is Grokking a Loss of Normal Hyperbolicity of the Interpolation Manifold?
• Abstract summary: In a fast-slow system, the manifold plays the role of a slow manifold.<n>We show that the sharp generalization transition does not collapse at the transition.<n>This is preliminary evidence against the bifurcation hypothesis.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: A recent line of work recasts the post-memorization phase of grokking as constrained optimization: once a network interpolates the training set, weight decay drives a slow drift along the zero-loss manifold toward lower norm. In the language of dynamical systems, this is a fast-slow system in which the interpolation manifold plays the role of a slow manifold. We ask a question that this framing makes natural but the existing literature does not address: is the sharp generalization transition a loss of normal hyperbolicity of that manifold: a fold- or bifurcation-like event in which a normal restoring direction goes flat? Or does the manifold stay uniformly attracting while generalization happens by smooth drift? We propose a simple, optimizer-agnostic diagnostic: the smallest nonzero singular value $σ_{\min}^{+}(\mathbf J)$ of the residual Jacobian, which, for the squared loss, equals the slowest normal restoring rate of the manifold. On a two-layer ReLU network trained to grok modular addition under squared loss, $σ_{\min}^{+}(\mathbf J)$ does not collapse at the transition; it is near zero only before memorization and attains its largest values during the transition. The result holds across five seeds, and the six smallest singular values behave identically; there is no subspace-local collapse either. This is preliminary evidence against the bifurcation hypothesis and in favor of the smooth-contraction picture. We are explicit that a single-setting, gradual-transition experiment under Adam optimizer does not prove the absence of a bifurcation; it constrains where one could hide.

Related papers

• Optimization Geometrodynamics: Variational Reduction and Interaction Curvature [0.0]
  We develop geometrodynamics as a variational theory of this hidden determinant pushforward.<n>For affine pre-reductions, the induced interaction curvature is the negative-semidefinite operator $-G*H-1G$.<n>We prove a global analytic bundle with closed totally geodesic fibers and a unique analytic nearest-controller section.
  arXiv  Detail & Related papers  (2026-07-07T18:42:00Z)
• Semidefinite Relaxations for Collision-Free Motion Planning [53.160637667144876]
  We focus on a point robot moving from start to goal through inmathbbRn constraints in terms of $C4 trajectories.<n>We demonstrate its effectiveness as a planner for minimum-snap continuous planning with $C4 trajectories.
  arXiv  Detail & Related papers  (2026-06-12T03:18:03Z)
• Is Spurious Correlation Removal Always Learnable? [56.28155520961125]
  Invariant learning can fail even when the structure is statistically identifiable.<n>We show a conditional computational barrier: under a black-box samplable supervised sparse recovery primitive, there exist emphamplable multi-dimensional environments.<n>Synthetic and real datasets illustrate the predicted gaps and transitions and motivate simple diversity diagnostics.
  arXiv  Detail & Related papers  (2026-06-11T05:49:43Z)
• Self-Normalized Martingales and Uniform Regret Bounds for Linear Regression [65.82017723631897]
  We show that scale-invariant upper bounds on self-normalized martingales are possible.<n>We derive a self-normalized concentration inequality free of the usual regularization penalties.
  arXiv  Detail & Related papers  (2026-05-02T22:39:00Z)
• $Z^2$-Sampling: Zero-Cost Zigzag Trajectories for Semantic Alignment in Diffusion Models [6.21141073537668]
  Implicit Z-Sampling proves that intermediate states can be algebraically annihilated via operator dualities.<n>$Z2$-Sampling couples implicit algebraic collapse with a dynamically cached Temporal Semantic Surrogate.
  arXiv  Detail & Related papers  (2026-04-26T05:16:54Z)
• Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold [0.0]
  Generalization requires the textitanisotropic rectification unique to adaptives, which directs noise into the tangent space of the solution.<n>This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a Spectral Gating'' mechanism that regulates the transition from generalization to memorization.
  arXiv  Detail & Related papers  (2026-03-16T16:18:18Z)
• Finite time path field theory and a new type of universal quantum spin chain quench behavior [0.0]
  We show that the difference between the sudden quench and a class of quenches with non-sudden switching on the perturbation vanishes exponentially with time.<n>We also show that analogous universal behavior exists in disorder quenches, and ultimately global ones.
  arXiv  Detail & Related papers  (2025-06-20T18:08:30Z)
• Exact dynamics of quantum dissipative $XX$ models: Wannier-Stark localization in the fragmented operator space [49.1574468325115]
  We find an exceptional point at a critical dissipation strength that separates oscillating and non-oscillating decay. We also describe a different type of dissipation that leads to a single decay mode in the whole operator subspace.
  arXiv  Detail & Related papers  (2024-05-27T16:11:39Z)
• Renormalization group for measurement and entanglement phase transitions [0.0]
  We analyze the renormalization-group (RG) flows of two effective Lagrangians. We show that the theory for the random tensor network formally possesses a dimensional reduction property analogous to that of the random-field Ising model.
  arXiv  Detail & Related papers  (2023-03-14T12:40:03Z)
• Complete positivity violation of the reduced dynamics in higher-order quantum adiabatic elimination [0.0]
  This paper discusses quantum adiabatic elimination, which is a model reduction technique for a composite Lindblad system. We present two examples where complete positivity of the reduced dynamics is violated with higher-order contributions.
  arXiv  Detail & Related papers  (2023-03-08T10:38:27Z)
• Agnostic Learning of Halfspaces with Gradient Descent via Soft Margins [92.7662890047311]
  We show that gradient descent finds halfspaces with classification error $tilde O(mathsfOPT1/2) + varepsilon$ in $mathrmpoly(d,1/varepsilon)$ time and sample complexity.
  arXiv  Detail & Related papers  (2020-10-01T16:48:33Z)
• Anisotropy-mediated reentrant localization [62.997667081978825]
  We consider a 2d dipolar system, $d=2$, with the generalized dipole-dipole interaction $sim r-a$, and the power $a$ controlled experimentally in trapped-ion or Rydberg-atom systems. We show that the spatially homogeneous tilt $beta$ of the dipoles giving rise to the anisotropic dipole exchange leads to the non-trivial reentrant localization beyond the locator expansion.
  arXiv  Detail & Related papers  (2020-01-31T19:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.