Procedural Pretraining: Warming Up Language Models with Abstract Data

• URL: http://arxiv.org/abs/2601.21725v1
• Date: Thu, 29 Jan 2026 13:48:43 GMT
• Title: Procedural Pretraining: Warming Up Language Models with Abstract Data
• Authors: Liangze Jiang, Zachary Shinnick, Anton van den Hengel, Hemanth Saratchandran, Damien Teney,
• Abstract summary: Pretraining directly on web-scale corpora is the de facto paradigm for building language models.<n>We study an alternative setting where the model is initially exposed to abstract structured data, as a means to ease the subsequent acquisition of rich semantic knowledge.<n>We specifically focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data.
• Score: 40.183555811204506
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining directly on web-scale corpora is the de facto paradigm for building language models. We study an alternative setting where the model is initially exposed to abstract structured data, as a means to ease the subsequent acquisition of rich semantic knowledge, much like humans learn simple logic and mathematics before higher reasoning. We specifically focus on procedural data, generated by formal languages and other simple algorithms, as such abstract data. We first diagnose the algorithmic skills that different forms of procedural data can improve, often significantly. For example, on context recall (Needle-in-a-haystack), the accuracy jumps from 10 to 98% when pretraining on Dyck sequences (balanced brackets). Second, we study how these gains are reflected in pretraining larger models (up to 1.3B). We find that front-loading as little as 0.1% procedural data significantly outperforms standard pretraining on natural language, code, and informal mathematics (C4, CodeParrot, and DeepMind-Math datasets). Notably, this procedural pretraining enables the models to reach the same loss value with only 55, 67, 86% of the original data. Third, we explore the mechanisms behind and find that procedural pretraining instils non-trivial structure in both attention and MLP layers. The former is particularly important for structured domains (e.g. code), and the latter for language. Finally, we lay a path for combining multiple forms of procedural data. Our results show that procedural pretraining is a simple, lightweight means to improving performance and accelerating language model pretraining, ultimately suggesting the promise of disentangling knowledge acquisition from reasoning in LLMs.

Related papers

• Approximating Language Model Training Data from Weights [70.08614275061689]
  We formalize the problem of data approximation from model weights and propose several baselines and metrics.<n>We develop a gradient-based approach that selects the highest-matching data from a large public text corpus.<n>Even when none of the true training data is known, our method is able to locate a small subset of public Web documents.
  arXiv  Detail & Related papers  (2025-06-18T15:26:43Z)
• Transformers Pretrained on Procedural Data Contain Modular Structures for Algorithmic Reasoning [40.84344912259233]
  We identify several beneficial forms of procedural data, together with specific algorithmic reasoning skills that improve in small transformers.<n>Our core finding is that different procedural rules instil distinct but complementary inductive structures in the model.<n>Most interestingly, the structures induced by multiple rules can be composed to jointly impart multiple capabilities.
  arXiv  Detail & Related papers  (2025-05-28T12:50:09Z)
• Pretraining Language Models to Ponder in Continuous Space [50.52734567589996]
  We introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.<n>We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
  arXiv  Detail & Related papers  (2025-05-27T03:47:33Z)
• Reasoning to Learn from Latent Thoughts [61.2395150828168]
  We show that explicitly modeling and inferring the emphlatent thoughts that underlie the text generation process can significantly improve pretraining data efficiency.<n>We show that a 1B LM can bootstrap its performance across at least three iterations and significantly outperform baselines trained on raw data.
  arXiv  Detail & Related papers  (2025-03-24T16:41:23Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models [29.367678364485794]
  We show how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. We show an improvement of 9% in average model accuracy compared to the baseline of continued training on the pretraining set.
  arXiv  Detail & Related papers  (2024-07-09T22:37:59Z)
• Investigating Pre-trained Language Models on Cross-Domain Datasets, a Step Closer to General AI [0.8889304968879164]
  We investigate the ability of pre-trained language models to generalize to different non-language tasks. The four pre-trained models that we used, T5, BART, BERT, and GPT-2 achieve outstanding results.
  arXiv  Detail & Related papers  (2023-06-21T11:55:17Z)
• SPDF: Sparse Pre-training and Dense Fine-tuning for Large Language Models [4.114555639014612]
  We show the benefits of using unstructured weight sparsity to train only a subset of weights during pre-training. We demonstrate that we can induce up to 75% sparsity into a 1.3B parameter GPT-3 XL model resulting in a 2.5x reduction in pre-training FLOPs.
  arXiv  Detail & Related papers  (2023-03-18T17:56:01Z)
• Memory Efficient Continual Learning for Neural Text Classification [10.70710638820641]
  We devise a method to perform text classification using pre-trained models on a sequence of classification tasks provided in sequence. We empirically demonstrate that our method requires significantly less model parameters compared to other state of the art methods. While our method suffers little forgetting, it retains a predictive performance on-par with state of the art but less memory efficient methods.
  arXiv  Detail & Related papers  (2022-03-09T10:57:59Z)
• bert2BERT: Towards Reusable Pretrained Language Models [51.078081486422896]
  We propose bert2BERT, which can effectively transfer the knowledge of an existing smaller pre-trained model to a large model. bert2BERT saves about 45% and 47% computational cost of pre-training BERT_BASE and GPT_BASE by reusing the models of almost their half sizes.
  arXiv  Detail & Related papers  (2021-10-14T04:05:25Z)
• Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting [66.45372974713189]
  We propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Experiments show that our method achieves state-of-the-art performance on the GLUE benchmark. We provide open-source RecAdam, which integrates the proposed mechanisms into Adam to facility the NLP community.
  arXiv  Detail & Related papers  (2020-04-27T08:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.