Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generation

• URL: http://arxiv.org/abs/2609.04484v1
• Date: Thu, 03 Sep 2026 21:07:43 GMT
• Title: Patterns of Priming in Production: Lexical, Semantic and Structural Alignment in Language Model Generation
• Abstract summary: We find that LMs are susceptible to structural priming, particularly in sentences that are semantically coherent.<n>We also observe that structural priming is boosted by lexico-semantic coherence.<n> Taken together, our evidence supports the view that structural priming in LMs operates across multiple levels of linguistic representation.
• Score: 12.118484518643038
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper investigates structural priming in language model (LM) production, examining how preceding structural context influences sentence completion. While prior work has demonstrated priming effects in comprehension of structural alternations, it remained unclear whether these persist in production, where, when generating, an LM samples from many possible continuations at each step. We address this question through a series of controlled sentence-completion experiments on dative constructions. In line with prior work, we find that LMs are susceptible to structural priming, particularly in sentences that are semantically coherent. In terms of priming magnitude, we find that while there is a greater relative increase of double-object datives against our baselines, in line with inverse frequency effects, there is a larger absolute increase in prepositional-objects, the more frequently produced construction. Finally, we not only observe that structural priming is boosted by lexico-semantic coherence, but that structurally primed completions display greater levels of lexico-semantic repetition. Taken together, our evidence supports the view that structural priming in LMs operates across multiple levels of linguistic representation, facilitating, and facilitated by syntactic, lexical, and semantic alignment. Code: https://github.com/the-context-lab/primedproduction.

Related papers

• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• From Static Structures to Ensembles: Studying and Harnessing Protein Structure Tokenization [15.864659611818661]
  Protein structure tokenization converts 3D structures into discrete or vectorized representations.<n>Despite many recent works on structure tokenization, the properties of the underlying discrete representations are not well understood.<n>We show that the successful utilization of structural tokens in a language model for structure prediction depends on using rich, pre-trained sequence embeddings.
  arXiv  Detail & Related papers  (2025-11-13T07:58:24Z)
• Probing Syntax in Large Language Models: Successes and Remaining Challenges [7.9494253785082405]
  It remains unclear whether structural and/or statistical factors systematically affect these syntactic representations.<n>We conduct an in-depth analysis of structural probes on three controlled benchmarks.
  arXiv  Detail & Related papers  (2025-08-05T08:41:14Z)
• QUDsim: Quantifying Discourse Similarities in LLM-Generated Text [70.22275200293964]
  We introduce an abstraction based on linguistic theories in Questions Under Discussion (QUD) and question semantics to help quantify differences in discourse progression.<n>We then use this framework to build $textbfQUDsim$, a similarity metric that can detect discursive parallels between documents.<n>Using QUDsim, we find that LLMs often reuse discourse structures (more so than humans) across samples, even when content differs.
  arXiv  Detail & Related papers  (2025-04-12T23:46:09Z)
• Linguistic Structure Induction from Language Models [1.8130068086063336]
  This thesis focuses on producing constituency and dependency structures from Language Models (LMs) in an unsupervised setting. I present a detailed study on StructFormer (SF) which retrofits a transformer architecture with a encoder network to produce constituency and dependency structures. I present six experiments to analyze and address this field's challenges.
  arXiv  Detail & Related papers  (2024-03-11T16:54:49Z)
• How Abstract Is Linguistic Generalization in Large Language Models? Experiments with Argument Structure [2.530495315660486]
  We investigate the degree to which pre-trained Transformer-based large language models represent relationships between contexts. We find that LLMs perform well in generalizing the distribution of a novel noun argument between related contexts. However, LLMs fail at generalizations between related contexts that have not been observed during pre-training.
  arXiv  Detail & Related papers  (2023-11-08T18:58:43Z)
• Grokking of Hierarchical Structure in Vanilla Transformers [72.45375959893218]
  We show that transformer language models can learn to generalize hierarchically after training for extremely long periods. intermediate-depth models generalize better than both very deep and very shallow transformers.
  arXiv  Detail & Related papers  (2023-05-30T04:34:13Z)
• Physics of Language Models: Part 1, Learning Hierarchical Language Structures [51.68385617116854]
  Transformer-based language models are effective but complex, and understanding their inner workings and reasoning mechanisms is a significant challenge.<n>We introduce a family of synthetic CFGs that produce hierarchical rules, capable of generating lengthy sentences.<n>We demonstrate that generative models like GPT can accurately learn and reason over CFG-defined hierarchies and generate sentences based on it.
  arXiv  Detail & Related papers  (2023-05-23T04:28:16Z)
• Does BERT really agree ? Fine-grained Analysis of Lexical Dependence on a Syntactic Task [70.29624135819884]
  We study the extent to which BERT is able to perform lexically-independent subject-verb number agreement (NA) on targeted syntactic templates. Our results on nonce sentences suggest that the model generalizes well for simple templates, but fails to perform lexically-independent syntactic generalization when as little as one attractor is present.
  arXiv  Detail & Related papers  (2022-04-14T11:33:15Z)
• Compositional Generalization Requires Compositional Parsers [69.77216620997305]
  We compare sequence-to-sequence models and models guided by compositional principles on the recent COGS corpus. We show structural generalization is a key measure of compositional generalization and requires models that are aware of complex structure.
  arXiv  Detail & Related papers  (2022-02-24T07:36:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.