How to Tame Grokking: Representation Geometry as a Control Signal

• URL: http://arxiv.org/abs/2607.11666v1
• Date: Mon, 13 Jul 2026 15:09:42 GMT
• Title: How to Tame Grokking: Representation Geometry as a Control Signal
• Abstract summary: Grokking is a phenomenon in which neural networks memorize training data and only later exhibit strong generalization after prolonged optimization.<n>We introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training.<n>In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grokking is a phenomenon in which neural networks initially memorize training data and only later exhibit strong generalization after prolonged optimization. Despite extensive recent study, the factors influencing the emergence and timing of grokking remain incompletely understood. We investigate the relationship between representation geometry and delayed generalization. We find that dimensionality collapse consistently precedes the onset of grokking in all evaluated settings. Motivated by these observations, we introduce Geometric Dimensionality Regularization (GeomDR), a simple spectral regularizer that modifies the effective dimensionality of hidden representations during training. Across modular addition, modular division, and permutation composition tasks, GeomDR consistently alters grokking dynamics and can substantially accelerate the onset of generalization depending on the intervention schedule and target dimensionality. In several settings, grokking is accelerated by up to 52 times relative to standard AdamW training. Similar qualitative effects are observed in both multilayer perceptrons and transformers. Together, these results suggest that representation geometry can serve as an effective control signal for grokking and provide evidence that geometric interventions offer a practical approach for studying and influencing delayed generalization in neural networks.

Related papers

• General Covariant Action Modeling: Constructing Generalized Manifolds via Spatio-Temporal Decoupling [48.559681359374146]
  Generalized Action Manifold (GAM) framework enforces general covariance through structural disentanglement.<n>GAM enables superior transfer and robustness capabilities, outperforming geometry-agnostic baselines.
  arXiv  Detail & Related papers  (2026-05-27T03:38:15Z)
• Monitoring Neural Training with Topology: A Footprint-Predictable Collapse Index [51.56484100374058]
  We propose an online, topology-aware monitor for evolving neural representations that couples Modular Morse Homology Maintenance (MMHM) with a composite Collapse Index (CI)<n>Instead of rebuilding complexes each epoch, we apply sparse edits at a fixed scale and maintain a discrete Morse matching, yielding fast, incremental updates.
  arXiv  Detail & Related papers  (2026-04-28T19:21:21Z)
• A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization [0.0]
  Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood.<n>We present a controlled study that systematically disentangles factors on modular addition (mod 97)<n>Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between stability and regularization.
  arXiv  Detail & Related papers  (2026-03-26T04:16:01Z)
• The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology [0.0]
  We study grokking - delayed generalization in Transformers trained on cyclic modular addition (Zp)<n>We identify two independent structural factors in standard Transformers: representational magnitude and data-dependent attention routing.
  arXiv  Detail & Related papers  (2026-03-05T14:41:01Z)
• GeoGNN: Quantifying and Mitigating Semantic Drift in Text-Attributed Graphs [59.61242815508687]
  Graph neural networks (GNNs) on text--attributed graphs (TAGs) encode node texts using pretrained language models (PLMs) and propagate these embeddings through linear neighborhood aggregation.<n>This work introduces a local PCA-based metric that measures the degree of semantic drift and provides the first quantitative framework to analyze how different aggregation mechanisms affect manifold structure.
  arXiv  Detail & Related papers  (2025-11-12T06:48:43Z)
• GeoAda: Efficiently Finetune Geometric Diffusion Models with Equivariant Adapters [61.51810815162003]
  We propose an SE(3)-equivariant adapter framework ( GeoAda) that enables flexible and parameter-efficient fine-tuning for controlled generative tasks.<n>GeoAda preserves the model's geometric consistency while mitigating overfitting and catastrophic forgetting.<n>We demonstrate the wide applicability of GeoAda across diverse geometric control types, including frame control, global control, subgraph control, and a broad range of application domains.
  arXiv  Detail & Related papers  (2025-07-02T18:44:03Z)
• Geometry-Editable and Appearance-Preserving Object Compositon [67.98806888489385]
  General object composition (GOC) aims to seamlessly integrate a target object into a background scene with desired geometric properties.<n>Recent approaches derive semantic embeddings and integrate them into advanced diffusion models to enable geometry-editable generation.<n>We introduce a Disentangled Geometry-editable and Appearance-preserving Diffusion model that first leverages semantic embeddings to implicitly capture desired geometric transformations.
  arXiv  Detail & Related papers  (2025-05-27T09:05:28Z)
• Beyond Progress Measures: Theoretical Insights into the Mechanism of Grokking [50.465604300990904]
  Grokking refers to the abrupt improvement in test accuracy after extended overfitting.<n>In this work, we investigate the grokking mechanism underlying the Transformer in the task of prime number operations.
  arXiv  Detail & Related papers  (2025-04-04T04:42:38Z)
• Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks [27.020990219204343]
  We investigate the influence of internal network structures on grokking.<n>We show that utilizing lottery tickets during the generalizing phase (termed grokked tickets) significantly reduces delayed generalization.<n>We find that grokked tickets exhibit periodic weight patterns, beneficial graph properties, and undergo rapid structural changes.
  arXiv  Detail & Related papers  (2023-10-30T11:58:44Z)
• On the geometry of generalization and memorization in deep neural networks [15.250162344382051]
  We study the structure of when and where memorization occurs in a deep network. All layers preferentially learn from examples which share features, and link this behavior to generalization performance. We find that memorization predominately occurs in the deeper layers, due to decreasing object' radius and dimension.
  arXiv  Detail & Related papers  (2021-05-30T19:07:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.