Structure-Specific Representational Priors Causally Control the Grokking Delay

• URL: http://arxiv.org/abs/2607.04333v3
• Date: Tue, 14 Jul 2026 13:00:28 GMT
• Title: Structure-Specific Representational Priors Causally Control the Grokking Delay
• Abstract summary: Grokking -- generalization long after training-set -- has been accelerated by structure-agnostic interventions.<n>We show it is causally the time to form the right representational structure -- decided at the level of features, not labels.<n>Because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Grokking -- generalization long after training-set interpolation -- has been accelerated by structure-agnostic interventions (gradient filtering, weight-norm clamping, geometric penalties). Whether the delay specifically measures the time to form task-structured representations has remained observational. We test it causally by injecting representational priors of varying content into a one-layer transformer learning modular addition, via a supervised-contrastive loss whose positives encode (i) the task's true structure ($(a+b) \bmod p$), (ii) a coherent-but-wrong sibling ($(a-b) \bmod p$), or (iii) a random partition -- all with identical loss form, strength, class sizes, and geometry. Whether generalization occurs follows a clean gradation: true 22/30 runs, sibling (same periodic features, wrong combination) 14/15, random (only memorizable) 0/20 (Fisher $p=1.3\times10^{-7}$). A weight-norm-matched control replaying the norm trajectory onto plain cross-entropy generalizes 0/15, ruling out the norm as mediator. Probes show structure formation precedes and predicts generalization in all runs. Only the true structure also accelerates grokking (up to $2.75\times$), but this is dose-dependent and bimodal. We then confirm the mechanism by prediction: because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator with a median $8.6\times$ speedup (up to $22\times$ on the fastest seeds, under 1000 epochs), growing monotonically as the norm is held lower; the residual stalls also vanish, though significant only pooled over the two mitigations run at both strengths ($0/40$ vs $6/20$, $p=7.7\times10^{-4}$), not per method. The grokking delay is, causally, the time to form the right representational structure -- decided at the level of features, not labels.

Related papers

• Quantifying the Memorization-to-Generalization Transition: Scaling Laws and Phase Structure in Grokking [0.0]
  We map the $0.8-to-generalization boundary across 384 configurations of two-hidden generalizations on modular arithmetic.<n>Data complexity is the dominant driver of regime transition, not model capacity.
  arXiv  Detail & Related papers  (2026-09-09T16:47:03Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking [0.0]
  We characterize what makes such a prior work, across four axes, in 188 new runs.<n>A coherent, learnable prior built from the wrong feature family blocks generalization like a random partition.<n>The prior is only needed early -- applied solely during the first 2000 epochs.
  arXiv  Detail & Related papers  (2026-07-14T13:06:44Z)
• Attention is Just Another Name for Coupling?: A Fast-Slow ODE Perspective on Hierarchical Pretraining [0.0]
  This paper asks whether a second, temporally slower coupling is fed back into the fast path through a zero-initialised gate-complements it.<n>The paper instantiates the fast-slow ODE formalism as a concrete neural network.
  arXiv  Detail & Related papers  (2026-06-15T13:54:15Z)
• Transformers Provably Learn to Internalize Chain-of-Thought [65.41010769606844]
  Implicit Chain-of-Thought (ICoT) trains models to internalize intermediate steps within their hidden states.<n>Log-ICoT removes thinking tokens one at a time, reducing the number of stages from linear in $k$ to logarithmic.<n>Experiments on multi-layer transformers confirm the theory and visualize how reasoning is progressively absorbed into deeper layers.
  arXiv  Detail & Related papers  (2026-05-27T15:17:06Z)
• Prism: Structural Symmetry Scanning via Duality-Constrained Laplacian Projection [0.0]
  We introduce textbfPrism, a framework for structural symmetry diagnosis in complex networks.<n>Prism computes the emphduality defect $(L,P) = |LP - PL|_F / |L|_F$ -- a scalar measuring how far the network deviates from structural self-consistency.
  arXiv  Detail & Related papers  (2026-05-18T04:20:05Z)
• Grokking as Structural Inference: Transformers Need Bayesian Lottery Tickets [0.0]
  We study why a Transformer that has memorized its training set wait thousands of steps before it generalizes.<n>We prove that this explaining-away delay can be bypassed by a KL-based structural intervention.<n>Experiments on algorithmic sequence tasks isolate structure from capacity and show that this Bayesian ticket matches or outperforms lottery-ticket transfer.
  arXiv  Detail & Related papers  (2026-05-15T09:46:36Z)
• Deep Reasoning in General Purpose Agents via Structured Meta-Cognition [58.185853639335896]
  We introduce an inference-time approach for constructing task-specific scaffolds through structured meta-reasoning.<n>We instantiate this approach in a general-purpose agent that distributes complex tasks across more controlled reasoning threads.
  arXiv  Detail & Related papers  (2026-05-12T01:21:37Z)
• Faster 3D Gaussian Splatting Convergence via Structure-Aware Densification [52.517763621139956]
  3D Gaussian Splatting has emerged as a powerful scene representation for real-time novel-view synthesis.<n>Standard adaptive density control relies on screen-space positional gradients.<n>We present a structure-aware densification framework for 3D Gaussian Splatting.
  arXiv  Detail & Related papers  (2026-04-30T15:37:20Z)
• A Systematic Empirical Study of Grokking: Depth, Architecture, Activation, and Regularization [0.0]
  Grokking the delayed transition from memorization to generalization in neural networks remains poorly understood.<n>We present a controlled study that systematically disentangles factors on modular addition (mod 97)<n>Our central finding is that grokking dynamics are not primarily determined by architecture, but by interactions between stability and regularization.
  arXiv  Detail & Related papers  (2026-03-26T04:16:01Z)
• The Geometric Inductive Bias of Grokking: Bypassing Phase Transitions via Architectural Topology [0.0]
  We study grokking - delayed generalization in Transformers trained on cyclic modular addition (Zp)<n>We identify two independent structural factors in standard Transformers: representational magnitude and data-dependent attention routing.
  arXiv  Detail & Related papers  (2026-03-05T14:41:01Z)
• Optimal Unconstrained Self-Distillation in Ridge Regression: Strict Improvements, Precise Asymptotics, and One-Shot Tuning [61.07540493350384]
  Self-distillation (SD) is the process of retraining a student on a mixture of ground-truth and the teacher's own predictions.<n>We show that for any prediction risk, the optimally mixed student improves upon the ridge teacher for every regularization level.<n>We propose a consistent one-shot tuning method to estimate $star$ without grid search, sample splitting, or refitting.
  arXiv  Detail & Related papers  (2026-02-19T17:21:15Z)
• Rational Transductors [30.916600771560933]
  We introduce emphRational Transductors, a dual-stream architecture that augments the Transformer with a matrix-valued recurrence.<n>By injecting rational state information into the attention mechanism via a emphDeep Rational Injection scheme, our framework strictly generalizes the expressive power of Transformers.
  arXiv  Detail & Related papers  (2026-02-07T16:01:10Z)
• Latency and Ordering Effects in Online Decisions [0.0]
  Online decision systems operate under delayed feedback and order-sensitive dynamics.<n>We package heterogeneous latency, noncommutativity, and implementation-gap effects into a single lower-bound statement.
  arXiv  Detail & Related papers  (2025-11-17T07:08:05Z)
• INC: An Indirect Neural Corrector for Auto-Regressive Hybrid PDE Solvers [61.84396402100827]
  We propose the Indirect Neural Corrector ($mathrmINC$), which integrates learned corrections into the governing equations.<n>$mathrmINC$ reduces the error amplification on the order of $t-1 + L$, where $t$ is the timestep and $L$ the Lipschitz constant.<n>We test $mathrmINC$ in extensive benchmarks, covering numerous differentiable solvers, neural backbones, and test cases ranging from a 1D chaotic system to 3D turbulence.
  arXiv  Detail & Related papers  (2025-11-16T20:14:28Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learning [53.68371566336254]
  We argue that the key to better performance lies in meaningful latent modality structures instead of perfect modality alignment. Specifically, we design 1) a deep feature separation loss for intra-modality regularization; 2) a Brownian-bridge loss for inter-modality regularization; and 3) a geometric consistency loss for both intra- and inter-modality regularization.
  arXiv  Detail & Related papers  (2023-03-10T14:38:49Z)
• DepGraph: Towards Any Structural Pruning [68.40343338847664]
  We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers. We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning. In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
  arXiv  Detail & Related papers  (2023-01-30T14:02:33Z)
• Coarsening the Granularity: Towards Structurally Sparse Lottery Tickets [127.56361320894861]
  Lottery ticket hypothesis (LTH) has shown that dense models contain highly sparseworks (i.e., winning tickets) that can be trained in isolation to match full accuracy. In this paper, we demonstrate the first positive result that a structurally sparse winning ticket can be effectively found in general. Specifically, we first "re-fill" pruned elements back in some channels deemed to be important, and then "re-group" non-zero elements to create flexible group-wise structural patterns.
  arXiv  Detail & Related papers  (2022-02-09T21:33:51Z)
• Towards Defending Multiple $\ell_p$-norm Bounded Adversarial Perturbations via Gated Batch Normalization [120.99395850108422]
  Existing adversarial defenses typically improve model robustness against individual specific perturbations. Some recent methods improve model robustness against adversarial attacks in multiple $ell_p$ balls, but their performance against each perturbation type is still far from satisfactory. We propose Gated Batch Normalization (GBN) to adversarially train a perturbation-invariant predictor for defending multiple $ell_p bounded adversarial perturbations.
  arXiv  Detail & Related papers  (2020-12-03T02:26:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.