A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization

• URL: http://arxiv.org/abs/2606.00230v1
• Date: Fri, 29 May 2026 18:04:52 GMT
• Title: A Pre-Training Analogue of Grokking in Language Models: Tracing Delayed Grammatical Generalization
• Abstract summary: Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs.<n>We propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training.
• Score: 11.118638230247951
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Grokking, the phenomenon in which neural networks generalize long after fitting their training data, has been studied in supervised settings on many epochs. LLM pre-training instead involves next-token prediction over an unlabeled corpus, with limited data repetition and no explicit train/validation split. To address this, we propose an exposure-based framework that enables the study of grokking-like dynamics during LLM pre-training. We ground our evaluation in BLiMP minimal pairs, which provide controlled grammatical contrasts. For every BLiMP minimal pair, we identify a critical phrase, the smallest continuous span that captures the grammatical contrast and the phenomenon-relevant context. Examples whose critical phrase appears in the pre-training window are assigned to the proxy-train split; the remaining examples are assigned to the proxy-validation split. Across five grammatical phenomena, we observe delayed generalization. Analyzing pre-training checkpoints before and after generalization shows that grammatical concept vectors become more predictive of grammatical acceptability and occupy a higher-dimensional subspace after generalization. We also find that attention from the critical token to the relevant context token is concentrated in a small number of heads.

Related papers

• BID: Boundary-Interior Decoding for Unsupervised Temporal Action Localization Pre-Trainin [13.273908640951252]
  We propose the first unsupervised pre-training framework that partitions a skeleton-based motion sequence into semantically meaningful pre-action segments. By fine-tuning our pre-training network with a small number of annotated data, we show results out-performing SOTA methods by a large margin.
  arXiv  Detail & Related papers  (2024-03-12T06:23:45Z)
• Enhancing Supervised Learning with Contrastive Markings in Neural Machine Translation Training [10.498938255717066]
  Supervised learning in Neural Machine Translation (NMT) typically follows a teacher forcing paradigm. We present a simple extension of standard maximum likelihood estimation by a contrastive marking objective. We show that training with contrastive markings yields improvements on top of supervised learning.
  arXiv  Detail & Related papers  (2023-07-17T11:56:32Z)
• Assessing Phrase Break of ESL speech with Pre-trained Language Models [6.635783609515407]
  This work introduces an approach to assessing phrase break in ESL learners' speech with pre-trained language models (PLMs) Different with traditional methods, this proposal converts speech to token sequences, and then leverages the power of PLMs.
  arXiv  Detail & Related papers  (2022-10-28T10:06:06Z)
• Sentence Representation Learning with Generative Objective rather than Contrastive Objective [86.01683892956144]
  We propose a novel generative self-supervised learning objective based on phrase reconstruction. Our generative learning achieves powerful enough performance improvement and outperforms the current state-of-the-art contrastive methods.
  arXiv  Detail & Related papers  (2022-10-16T07:47:46Z)
• On the Viability of Monocular Depth Pre-training for Semantic Segmentation [48.29060171161375]
  We study whether pre-training on geometric tasks is viable for downstream transfer to semantic tasks. We find that monocular depth is a viable form of pre-training for semantic segmentation, validated by improvements over common baselines.
  arXiv  Detail & Related papers  (2022-03-26T04:27:28Z)
• An Explanation of In-context Learning as Implicit Bayesian Inference [117.19809377740188]
  We study the role of the pretraining distribution on the emergence of in-context learning. We prove that in-context learning occurs implicitly via Bayesian inference of the latent concept. We empirically find that scaling model size improves in-context accuracy even when the pretraining loss is the same.
  arXiv  Detail & Related papers  (2021-11-03T09:12:33Z)
• The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design [34.900425311720795]
  We show that pretrained NLM can model stronger dependencies between text segments that appeared in the same training example, than it can between text segments that appeared in different training examples. We propose "kNN-Pretraining": we show that including semantically related non-neighboring sentences in the same pretraining example yields improved sentence representations and open domain question answering abilities.
  arXiv  Detail & Related papers  (2021-10-09T11:05:16Z)
• Token-wise Curriculum Learning for Neural Machine Translation [94.93133801641707]
  Existing curriculum learning approaches to Neural Machine Translation (NMT) require sufficient sampling amounts of "easy" samples from training data at the early training stage. We propose a novel token-wise curriculum learning approach that creates sufficient amounts of easy samples. Our approach can consistently outperform baselines on 5 language pairs, especially for low-resource languages.
  arXiv  Detail & Related papers  (2021-03-20T03:57:59Z)
• COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretraining [59.169836983883656]
  COCO-LM is a new self-supervised learning framework that pretrains Language Models by COrrecting challenging errors and COntrasting text sequences. COCO-LM employs an auxiliary language model to mask-and-predict tokens in original text sequences. Our analyses reveal that COCO-LM's advantages come from its challenging training signals, more contextualized token representations, and regularized sequence representations.
  arXiv  Detail & Related papers  (2021-02-16T22:24:29Z)
• Narrative Incoherence Detection [76.43894977558811]
  We propose the task of narrative incoherence detection as a new arena for inter-sentential semantic understanding. Given a multi-sentence narrative, decide whether there exist any semantic discrepancies in the narrative flow.
  arXiv  Detail & Related papers  (2020-12-21T07:18:08Z)
• Text Classification with Few Examples using Controlled Generalization [58.971750512415134]
  Current practice relies on pre-trained word embeddings to map words unseen in training to similar seen ones. Our alternative begins with sparse pre-trained representations derived from unlabeled parsed corpora. We show that a feed-forward network over these vectors is especially effective in low-data scenarios.
  arXiv  Detail & Related papers  (2020-05-18T06:04:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.