Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters

• URL: http://arxiv.org/abs/2607.05104v1
• Date: Mon, 06 Jul 2026 13:58:31 GMT
• Title: Grokking Is Conditional and Fragile: A Fully-Tractable, Multi-Seed Study at 12K Parameters
• Abstract summary: We study grokking as a multi-seed rate rather than a single outcome.<n>In this fully-tractable regime grokking is a conditional, fragile phase transition.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grokking--the delayed onset of generalization long after a network has fit its training set--is usually studied in models too large to read completely and reported from single training runs. We instead study a publicly released ~11,856-parameter Llama-style transformer (Glimmer-1-Base) on modular arithmetic, small enough to enumerate its weights, attention, and full input--output map, and we measure grokking as a multi-seed rate rather than a single outcome. In this fully-tractable regime grokking is a conditional, fragile phase transition. It is gated by training-set coverage, whose threshold tracks output cardinality (the modulus) more than task structure, an ordering that holds above the transition and across a ten-fold change in domain size. Weight decay reproduces the Omnigrok inverted-U at 12K parameters, a positive control on the rate measurement. Grokking also sits on a numerical knife-edge: two perturbations of the floating-point environment--CPU thread count (reduction order) and CPU-versus-GPU execution--each flip a minority of same-seed outcomes without a detectable shift in the aggregate rate. Decomposition into sub-task specialists helps chiefly by making coverage cheap rather than by adding supervision. Methodologically, multi-seed control under a fixed numerical environment overturns three dramatic single-run narratives in our own data, each a seed confound. The unit of evidence for grokking must therefore be a multi-seed rate under a pinned numerical environment, checked where possible against a direct reading of the model.

Related papers

• ActMap: Single-Pass Uncertainty Quantification from Generation-Time Activation Maps [1.7403133838762448]
  ActMap is a representation that compresses the generation-time hidden-state trajectory into a fixed $12times32times128$ of temporal-statistic channels.<n>A lightweight classifier, instantiated as a compact Vision Transformer, reads an estimated correctness probability from each map in a fraction of a millisecond.
  arXiv  Detail & Related papers  (2026-09-10T13:04:24Z)
• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• The Cost of Adaptivity: Matching Lower Bounds Across Learning Problems [1.0742675209112622]
  We formalize nuisance adaptation via a slice-normalized minimax ratio.<n>We also define the cost of expanding from one preannounced Gaussian query to arbitrary post-hoc inspection.
  arXiv  Detail & Related papers  (2026-08-09T17:19:11Z)
• Denoising growth complexity: Data geometry and certified schedules for diffusion sampling [14.877292137182422]
  We show how the $mathsfDGC$ increments lead to a simple and explicit bound on the KL error of an Euler scheme applied to the innovations representation.<n>We derive KL sampling guarantees for optimized stepsize schedules, both in a simpler single-block setting and in a more refined $K$-block setting.
  arXiv  Detail & Related papers  (2026-07-28T21:26:16Z)
• At-Grok Is Not Converged:A Measurement-Validity Audit for Grokking Representation Metrics [0.0]
  We show that compression lags the accuracy transition by an amount on the time-to-grok, at least 10,000 steps, rather than coinciding with it.<n>We package this as an audit that separates onset from compression flags, censoring, excludes cells that never fully generalize, and checks that the reference floor has plateaued.
  arXiv  Detail & Related papers  (2026-07-07T14:44:18Z)
• Closed-Form Spectral Regularization for Multi-Task Model Merging [96.82449201305234]
  Model merging combines several independently fine-tuned experts into a single multi-task model without any training data.<n>State-of-the-art merging methods formulate merging as a layer-wise interference problem.<n>We propose SWUDI, a closed-form method that combines a soft exponential filter, which matches the gradient-flow trajectory of iterative descent.
  arXiv  Detail & Related papers  (2026-06-05T14:00:47Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Dimensional Criticality at Grokking Across MLPs and Transformers [2.652953665748039]
  Abrupt transitions between distinct dynamical regimes are a hallmark of complex systems.<n>We introduce textbfTDU--OFC (Thresholded Diffusion Update--Olami-Feder-Christensen), an offline avalanche probe.<n>Across Transformers trained on modular addition and generalizations trained on XOR, we discover a localized crossing of the diffusion baseline $D=1$.
  arXiv  Detail & Related papers  (2026-04-06T13:43:20Z)
• Grokking as a Variance-Limited Phase Transition: Spectral Gating and the Epsilon-Stability Threshold [0.0]
  Generalization requires the textitanisotropic rectification unique to adaptives, which directs noise into the tangent space of the solution.<n>This work analyzes AdamW dynamics on modular arithmetic tasks, revealing a Spectral Gating'' mechanism that regulates the transition from generalization to memorization.
  arXiv  Detail & Related papers  (2026-03-16T16:18:18Z)
• SurgFormer: Scalable Learning of Organ Deformation with Resection Support and Real-Time Inference [14.49835131603816]
  We introduce SurgFormer, a multiresolution gated transformer for data driven soft tissue simulation on volumetric meshes.<n>High fidelity biomechanical solvers are often too costly for interactive use, so we train SurgFormer on solver generated data to predict nodewise displacement fields.<n>SurgFormer achieves strong accuracy with favorable efficiency, making it a practical backbone for both tasks.
  arXiv  Detail & Related papers  (2026-03-06T18:29:47Z)
• Late-Stage Generalization Collapse in Grokking: Detecting anti-grokking with Weightwatcher [1.6615337656760856]
  emphMemorization in neural networks lacks a precise operational definition and is often inferred from the grokking regime.<n>We identify a previously unreported third phase of grokking in this training regime: emphanti-grokking, a late-stage collapse of generalization.
  arXiv  Detail & Related papers  (2026-02-02T22:09:14Z)
• Gated Differentiable Working Memory for Long-Context Language Modeling [80.27483324685434]
  We propose Gdwm (Gated Differentiable Working Memory), a framework that introduces a write controller to gate the consolidation process.<n>Experiments on ZeroSCROLLS and LongBench v2 demonstrate that Gdwm achieves comparable or superior performance with 4$times$ fewer gradient steps than uniform baselines.
  arXiv  Detail & Related papers  (2026-01-19T10:00:33Z)
• MLPMoE: Zero-Shot Architectural Metamorphosis of Dense LLM MLPs into Static Mixture-of-Experts [0.0]
  Large Language Models (LLMs) are predominantly deployed as dense transformers, where every parameter in every feed-forward block is activated for every token.<n>Recent upcycling methods such as MoEfication, CMoE, ToMoE, and MoORE reveal that much of the useful computation lives in sparse, semi-modular substructures inside dense feed-forward networks.<n>This paper introducesMoE (MLP-Experts), a training-free transformation that restructures the dense in transformer blocks into a static, high-cardinality mixture of experts.
  arXiv  Detail & Related papers  (2025-11-26T06:14:26Z)
• DiffGRM: Diffusion-based Generative Recommendation Model [63.35379395455103]
  Generative recommendation (GR) is an emerging paradigm that represents each item via a tokenizer as an n-digit semantic ID (SID)<n>We propose DiffGRM, a diffusion-based GR model that replaces the autoregressive decoder with a masked discrete diffusion model (MDM)<n> Experiments show consistent gains over strong generative and discriminative recommendation baselines on multiple datasets.
  arXiv  Detail & Related papers  (2025-10-21T03:23:32Z)
• Semi-DETR: Semi-Supervised Object Detection with Detection Transformers [105.45018934087076]
  We analyze the DETR-based framework on semi-supervised object detection (SSOD) We present Semi-DETR, the first transformer-based end-to-end semi-supervised object detector. Our method outperforms all state-of-the-art methods by clear margins.
  arXiv  Detail & Related papers  (2023-07-16T16:32:14Z)
• Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing [113.52575069030192]
  Big data, including applications with high security requirements, are often collected and stored on multiple heterogeneous devices, such as mobile devices, drones and vehicles. Due to the limitations of communication costs and security requirements, it is of paramount importance to extract information in a decentralized manner instead of aggregating data to a fusion center. We consider the problem of learning model parameters in a multi-agent system with data locally processed via distributed edge nodes. A class of mini-batch alternating direction method of multipliers (ADMM) algorithms is explored to develop the distributed learning model.
  arXiv  Detail & Related papers  (2020-10-02T10:41:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.