What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking

• URL: http://arxiv.org/abs/2607.12735v1
• Date: Tue, 14 Jul 2026 13:06:44 GMT
• Title: What Makes a Representational Prior Work? Feature Families, Label-Free Invariances, and Critical Windows in Grokking
• Abstract summary: We characterize what makes such a prior work, across four axes, in 188 new runs.<n>A coherent, learnable prior built from the wrong feature family blocks generalization like a random partition.<n>The prior is only needed early -- applied solely during the first 2000 epochs.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Companion work showed the grokking delay is causally the time to form task-structured representations, injectable via a contrastive prior. Here we characterize what makes such a prior work, across four axes, in 188 new runs. Content: a coherent, learnable prior built from the wrong feature family (magnitude bands) blocks generalization like a random partition (1/15 vs 0/20 grok; $p=0.43$ between them), confirming the companion's prediction that priors act at the level of the circuit's features. Supervision: a fully label-free invariance prior -- positives are commuted pairs $(a,b)\sim(b,a)$ only -- generalizes in 15/15 runs at a median $2.7\times$ speedup, more reliably than the label-supervised prior itself ($p=0.038$), and combined with a weight-norm clamp yields the strongest method we test (median $17\times$, 5/5) -- strongest meaning reliably fast: plain cross-entropy with a clamp matches this speed only at the exact critical norm, while the prior keeps it fast across the entire clamp range. Timing: the prior is only needed early -- applied solely during the first 2000 epochs (4% of budget) it generalizes 10/10 at $2.7\times$, beating continuous application (8/10, $1.25\times$) and a duration-matched later window ($2.1\times$). Setting: the dissociation replicates on modular multiplication and across depths and normalization variants, and a clamp sweep quantifies the companion's central claim: structure injection flattens the weight-norm delay-law exponent about 17-fold (plain cross-entropy slows $31\times$ per +10 norm units, a lower bound as higher cells are censored, versus $1.22\times$ with the prior). Honest boundary: tasks that generalize before memorizing have no delay to control. Feature-family alignment decides whether a prior permits generalization; invariance content suffices for acceleration without labels; a brief early window captures nearly all of the benefit.

Related papers

• Structure-Specific Representational Priors Causally Control the Grokking Delay [0.0]
  Grokking -- generalization long after training-set -- has been accelerated by structure-agnostic interventions.<n>We show it is causally the time to form the right representational structure -- decided at the level of features, not labels.<n>Because the acceleration is gated by a weight-norm side-effect, clamping the norm during training yields a reliable, standalone accelerator.
  arXiv  Detail & Related papers  (2026-07-05T14:32:26Z)
• Attention is Just Another Name for Coupling?: A Fast-Slow ODE Perspective on Hierarchical Pretraining [0.0]
  This paper asks whether a second, temporally slower coupling is fed back into the fast path through a zero-initialised gate-complements it.<n>The paper instantiates the fast-slow ODE formalism as a concrete neural network.
  arXiv  Detail & Related papers  (2026-06-15T13:54:15Z)
• When Do LLM Agents Treat Surface Noise Differently from Semantic Noise? A 68-Cell Measurement Study with a Held-Out Trace-Level Validation [9.055086193088083]
  We document an empirical phenomenon in chain-of-thought and ReAct agents driven by ten large language models.<n>Mean-bearing perturbations alter final answers more often than presentation perturbations of comparable severity.
  arXiv  Detail & Related papers  (2026-05-25T15:57:11Z)
• Prism: Structural Symmetry Scanning via Duality-Constrained Laplacian Projection [0.0]
  We introduce textbfPrism, a framework for structural symmetry diagnosis in complex networks.<n>Prism computes the emphduality defect $(L,P) = |LP - PL|_F / |L|_F$ -- a scalar measuring how far the network deviates from structural self-consistency.
  arXiv  Detail & Related papers  (2026-05-18T04:20:05Z)
• Fast Rates for Offline Contextual Bandits with Forward-KL Regularization under Single-Policy Concentrability [54.40598524756038]
  emphKullback-Leibler (KL) regularization is ubiquitous in reinforcement learning algorithms.<n>Recent studies have demonstrated $-1$-type fast rates for decision making under reverse KL regularization.<n>We take the first step towards addressing this problem via a streamlined analysis of forward-KL-regularized offline CBs.
  arXiv  Detail & Related papers  (2026-05-09T23:17:46Z)
• Block-Wise Differentiable Sinkhorn Attention: Tail-Refinement Gradients with a Gap-Aware Dustbin Bridge [0.0]
  We study long-context balanced optimal transport attention on TPU hardware through a stopped-base, fixed-depth tail-refinement surrogate.<n>We provide a local surrogate-bias bound, an a posteriori bias certificate, and a projective contraction certificate for strictly positive active blocks.<n>On synthetic masked problems, the optimized matches exact autodiff of the same centered surrogate to within $10-5$--$10-10$.
  arXiv  Detail & Related papers  (2026-04-28T19:01:11Z)
• Prism-$Δ$: Differential Subspace Steering for Prompt Highlighting in Large Language Models [85.18197548789291]
  We propose PRISM-$$ (Projection-based Relevance-Informed Steering Method) to maximize discriminative energy while eliminating shared directions.<n> PRISM-$$ matches or exceeds the best existing method on 19 of 20 configurations, with relative gains up to +10.6%, while halving the cost of steering.<n> PRISM-$$ is compatible with FlashAttention and adds negligible memory overhead.
  arXiv  Detail & Related papers  (2026-03-11T12:24:45Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Scale-Invariant Regret Matching and Online Learning with Optimal Convergence: Bridging Theory and Practice in Zero-Sum Games [60.871651115241406]
  A considerable chasm has been looming for decades between theory and practice in zero-sum game solving through first-order methods.<n>We propose a new scale-invariance and parameter-free variant of PRM$+$, which we call IREG-PRM$+$.<n>We show that it achieves $T-1/2$ best-iterate and $T-1$ optimal convergence guarantees, while also being on par with PRM$+$ on benchmark games.
  arXiv  Detail & Related papers  (2025-10-06T00:33:20Z)
• Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction [57.93371273485736]
  We consider a centralized distributed learning setup where all workers jointly find an unbiased bound LDeltaepsilon2,$ better poly-logarithmically in $n$, even in the homogeneous (i.i.d.) case, where all workers access the same distribution.
  arXiv  Detail & Related papers  (2025-06-30T13:27:39Z)
• From Continual Learning to SGD and Back: Better Rates for Continual Linear Models [50.11453013647086]
  We analyze the forgetting, i.e., loss on previously seen tasks, after $k$ iterations.<n>We develop novel last-iterate upper bounds in the realizable least squares setup.<n>We prove for the first time that randomization alone, with no task repetition, can prevent catastrophic in sufficiently long task sequences.
  arXiv  Detail & Related papers  (2025-04-06T18:39:45Z)
• Sharper Convergence Guarantees for Asynchronous SGD for Distributed and Federated Learning [77.22019100456595]
  We show a training algorithm for distributed computation workers with varying communication frequency. In this work, we obtain a tighter convergence rate of $mathcalO!!!(sigma2-2_avg!! . We also show that the heterogeneity term in rate is affected by the average delay within each worker.
  arXiv  Detail & Related papers  (2022-06-16T17:10:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.