Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility

• URL: http://arxiv.org/abs/2608.03930v1
• Date: Tue, 04 Aug 2026 17:02:34 GMT
• Title: Logic Before Language: Pre-pretraining on Formal Derivations Fosters Skill Acquisition and Compressibility
• Abstract summary: Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition.<n>We show that logic pre-pretraining substantially accelerates skill acquisition in LMs.<n>Mechanistically, formal derivations induce persistent structural reorganization.
• Score: 41.15091250761469
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-pretraining language models (LMs) on symbolic data can accelerate and improve natural language acquisition. However, existing pre-pretraining tasks, such as Dyck and procedural algorithms, rely on narrow primitives that fail to capture the expressive capacity of natural language. Moreover, prior studies remain restricted to relatively small token budgets, offering limited insight into skill emergence and representational dynamics. To address these limitations, we propose logic pre-pretraining (Logic-PPT) as a principled initialization strategy, leveraging formal derivations to impart richer structural and linguistic biases. Formal derivations require abstract mechanisms that are central to natural language, simultaneously binding variables, connecting quantifiers and relational dependencies, and composing predicate-argument structures over long contexts. Scaling our evaluation to a 100B-token regime, logic pre-pretraining substantially accelerates skill acquisition in LMs, achieving 80\% accuracy on linguistic tasks with 36B fewer tokens than standard initialization, and outperforming alternative pre-pretraining baselines. Mechanistically, formal derivations induce persistent structural reorganization, distinctively characterized by a lower-rank, spectrally concentrated representation space. Crucially, we show that this internal geometry enables improved model compressibility via pruning, matching the dense baseline performance even at $\approx$33\% sparsity.

Related papers

• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• Small Initialization Matters for Large Language Models [28.726627416564842]
  We show that parameter initialization is a gene-like determinant of training and, in particular, of model capacity.<n>Reducing the scale consistently improves pretraining, with the largest gains on reasoning-demanding tasks.<n>We identify two widely used empirical settings that restrain the advantage of small initialization, and show how relaxing them restores favorable scaling.
  arXiv  Detail & Related papers  (2026-06-16T13:53:48Z)
• Language Acquisition Device in Large Language Models [20.65039123567166]
  We propose pre-pretraining on MP-STRUCT, a formal language whose strings encode hierarchical composition, feature-based dependencies, and long-distance displacement via MERGE, AGREE, and MOVE.<n>We find that MP-STRUCT CORE outperforms $k$-Shuffle Dyck despite not being definable in C-RASP.<n>We show that functional landmarks, which reduce dependency resolution ambiguity, are a key driver.
  arXiv  Detail & Related papers  (2026-05-16T02:13:32Z)
• Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models [82.79223371188756]
  Chain-of-Thought (CoT) prompting has advanced task-solving capabilities in natural language processing with large language models.<n>Applying CoT to non-natural language domains, such as protein and RNA language models, is not yet possible.<n>We introduce pretraining, for the first time in a biological sequence model, which enables the model to engage in intermediate reasoning.
  arXiv  Detail & Related papers  (2025-12-24T05:25:17Z)
• CSE-SFP: Enabling Unsupervised Sentence Representation Learning via a Single Forward Pass [3.0566617373924325]
  Recent advances in pre-trained language models (PLMs) have driven remarkable progress in this field.<n>We propose CSE-SFP, an innovative method that exploits the structural characteristics of generative models.<n>We show that CSE-SFP not only produces higher-quality embeddings but also significantly reduces both training time and memory consumption.
  arXiv  Detail & Related papers  (2025-05-01T08:27:14Z)
• HYPEROFA: Expanding LLM Vocabulary to New Languages via Hypernetwork-Based Embedding Initialization [50.27950279695363]
  Many pre-trained language models (PLMs) exhibit suboptimal performance on mid- and low-resource languages.<n>A common strategy to address this is to introduce new tokens specific to the target languages, initialize their embeddings, and apply continual pre-training on target-language data.<n>We propose HYPEROFA, a hypernetwork-based approach for more adaptive token embedding.
  arXiv  Detail & Related papers  (2025-04-21T19:40:32Z)
• Between Circuits and Chomsky: Pre-pretraining on Formal Languages Imparts Linguistic Biases [47.920937001420505]
  Pretraining language models on formal language can improve their acquisition of natural language.<n>We show that languages capturing hierarchical dependencies indeed enable language models to achieve lower loss on natural language.<n>We also give mechanistic evidence of transfer from formal to natural language.
  arXiv  Detail & Related papers  (2025-02-26T15:55:55Z)
• Implicit Language Models are RNNs: Balancing Parallelization and Expressivity [4.332158627306896]
  State-space models (SSMs) and transformers dominate the language modeling landscape.<n>We propose implicit SSMs, which implement a transformation until convergence to a fixed point.<n>Our approach demonstrates superior state-tracking capabilities on regular languages, surpassing transformers and SSMs.
  arXiv  Detail & Related papers  (2025-02-10T19:59:31Z)
• Leap-Of-Thought: Teaching Pre-Trained Models to Systematically Reason Over Implicit Knowledge [96.92252296244233]
  Large pre-trained language models (LMs) acquire some reasoning capacity, but this ability is difficult to control. We show that LMs can be trained to reliably perform systematic reasoning combining both implicit, pre-trained knowledge and explicit natural language statements. Our work paves a path towards open-domain systems that constantly improve by interacting with users who can instantly correct a model by adding simple natural language statements.
  arXiv  Detail & Related papers  (2020-06-11T17:02:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.