Grokking From Abstraction to Intelligence

• URL: http://arxiv.org/abs/2603.29262v1
• Date: Tue, 31 Mar 2026 04:57:12 GMT
• Title: Grokking From Abstraction to Intelligence
• Abstract summary: Grokking in modular arithmetic has established itself as the quintessential fruit fly experiment.<n>We propose that grokking originates from a spontaneous simplification of internal model structures governed by the principle of parsimony.
• Score: 15.03232023785853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grokking in modular arithmetic has established itself as the quintessential fruit fly experiment, serving as a critical domain for investigating the mechanistic origins of model generalization. Despite its significance, existing research remains narrowly focused on specific local circuits or optimization tuning, largely overlooking the global structural evolution that fundamentally drives this phenomenon. We propose that grokking originates from a spontaneous simplification of internal model structures governed by the principle of parsimony. We integrate causal, spectral, and algorithmic complexity measures alongside Singular Learning Theory to reveal that the transition from memorization to generalization corresponds to the physical collapse of redundant manifolds and deep information compression, offering a novel perspective for understanding the mechanisms of model overfitting and generalization.

Related papers

• CDFM: Towards a General-Purpose Causal Discovery Foundation Model [62.32138962508216]
  Causal discovery is the process of recovering underlying causal structures from observational data.<n>We formulate the Causal Discovery Foundation Model (CDFM) as a unified, general-purpose framework for zero-shot structural inference.<n>CDFM consistently outperforms traditional algorithms, driving a paradigm shift toward a general-purpose causal discovery foundation model.
  arXiv  Detail & Related papers  (2026-07-13T12:58:28Z)
• Pointwise Generalization in Deep Neural Networks [24.565019622696664]
  We identify the structural properties and mathematical principles that explain the tractability of deep networks.<n>We show that deep networks are mathematically tractable in practical regimes and that their generalization is sharply explained by pointwise, feature-spectrum-aware complexity.
  arXiv  Detail & Related papers  (2026-05-18T16:09:06Z)
• Rethinking Generalization in Graph Neural Networks: A Structural Complexity Perspective [41.727617218821386]
  Graph neural networks (GNNs) have emerged as a fundamental tool for learning from graph-structured data.<n>We ask: does the graph structure actually influence generalization, and if so, by how much?<n>To answer the first question and validate our intuition, we theoretically prove that incorporating more edges into the prediction process transforms the input representations to be overly accommodating to the output model, thereby inducing overfitting.<n>To address the second question, we formulate a structural complexity measure based on the number of effective edges and derive a Rademacher complexity-based generalization bound.
  arXiv  Detail & Related papers  (2026-05-13T14:32:46Z)
• Large Vision-Language Models Get Lost in Attention [51.851592109135716]
  We propose a unified framework grounded in information theory and geometry to quantify the geometric and entropic nature of residual updates.<n>Applying this unified framework reveals a fundamental functional decoupling: Attention acts as a subspace-language operator focused on reconfiguration, whereas FFNs serve as subspace-language operators driving semantic innovation.
  arXiv  Detail & Related papers  (2026-05-07T04:45:52Z)
• The Causal Round Trip: Generating Authentic Counterfactuals by Eliminating Information Loss [4.166536642958902]
  We introduce BELM-MDCM, the first diffusion-based framework engineered to be causally sound by eliminating the Structural Reconstruction Error (SRE)<n>Our work reconciles the power of modern generative models with the rigor of classical causal theory.
  arXiv  Detail & Related papers  (2025-11-07T13:37:23Z)
• A Mathematical Explanation of Transformers for Large Language Models and GPTs [6.245431127481903]
  We propose a novel continuous framework that interprets the Transformer as a discretization of a structured integro-differential equation.<n>Within this formulation, the self-attention mechanism emerges naturally as a non-local integral operator.<n>Our approach extends beyond previous theoretical analyses by embedding the entire Transformer operation in continuous domains.
  arXiv  Detail & Related papers  (2025-10-05T01:16:08Z)
• Learning Time-Aware Causal Representation for Model Generalization in Evolving Domains [50.66049136093248]
  We develop a time-aware structural causal model (SCM) that incorporates dynamic causal factors and the causal mechanism drifts.<n>We show that our method can yield the optimal causal predictor for each time domain.<n>Results on both synthetic and real-world datasets exhibit that SYNC can achieve superior temporal generalization performance.
  arXiv  Detail & Related papers  (2025-06-21T14:05:37Z)
• NeuralGrok: Accelerate Grokking by Neural Gradient Transformation [54.65707216563953]
  We propose NeuralGrok, a gradient-based approach that learns an optimal gradient transformation to accelerate generalization of transformers in arithmetic tasks.<n>Our experiments demonstrate that NeuralGrok significantly accelerates generalization, particularly in challenging arithmetic tasks.<n>We also show that NeuralGrok promotes a more stable training paradigm, constantly reducing the model's complexity.
  arXiv  Detail & Related papers  (2025-04-24T04:41:35Z)
• On the Role of Information Structure in Reinforcement Learning for Partially-Observable Sequential Teams and Games [55.2480439325792]
  In a sequential decision-making problem, the information structure is the description of how events in the system occurring at different points in time affect each other. By contrast, real-world sequential decision-making problems typically involve a complex and time-varying interdependence of system variables. We formalize a novel reinforcement learning model which explicitly represents the information structure.
  arXiv  Detail & Related papers  (2024-03-01T21:28:19Z)
• PAC-Chernoff Bounds: Understanding Generalization in the Interpolation Regime [6.645111950779666]
  This paper introduces a distribution-dependent PAC-Chernoff bound that exhibits perfect tightness for interpolators.<n>We present a unified theoretical framework revealing why certain interpolators show an exceptional generalization, while others falter.
  arXiv  Detail & Related papers  (2023-06-19T14:07:10Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• Abstract Interpretation for Generalized Heuristic Search in Model-Based Planning [50.96320003643406]
  Domain-general model-based planners often derive their generality by constructing searchs through the relaxation of symbolic world models. We illustrate how abstract interpretation can serve as a unifying framework for these abstractions, extending the reach of search to richer world models. Theses can also be integrated with learning, allowing agents to jumpstart planning in novel world models via abstraction-derived information.
  arXiv  Detail & Related papers  (2022-08-05T00:22:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.