Do Language Models Exhibit Human-like Structural Priming Effects?

• URL: http://arxiv.org/abs/2406.04847v2
• Date: Tue, 17 Sep 2024 15:17:36 GMT
• Title: Do Language Models Exhibit Human-like Structural Priming Effects?
• Authors: Jaap Jumelet, Willem Zuidema, Arabella Sinclair,
• Abstract summary: We investigate which linguistic factors play an important role in influencing language model predictions. We make use of the structural priming paradigm, where recent exposure to a structure facilitates processing of the same structure. We show that these effects can be explained via the inverse frequency effect, known in human priming, where rarer elements within a prime increase priming effects.
• Score: 3.4435563735186747
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We explore which linguistic factors -- at the sentence and token level -- play an important role in influencing language model predictions, and investigate whether these are reflective of results found in humans and human corpora (Gries and Kootstra, 2017). We make use of the structural priming paradigm, where recent exposure to a structure facilitates processing of the same structure. We don't only investigate whether, but also where priming effects occur, and what factors predict them. We show that these effects can be explained via the inverse frequency effect, known in human priming, where rarer elements within a prime increase priming effects, as well as lexical dependence between prime and target. Our results provide an important piece in the puzzle of understanding how properties within their context affect structural prediction in language models.

Related papers

• Probing Syntax in Large Language Models: Successes and Remaining Challenges [7.9494253785082405]
  It remains unclear whether structural and/or statistical factors systematically affect these syntactic representations.<n>We conduct an in-depth analysis of structural probes on three controlled benchmarks.
  arXiv  Detail & Related papers  (2025-08-05T08:41:14Z)
• Why Do Speech Language Models Fail to Generate Semantically Coherent Outputs? A Modality Evolving Perspective [23.49276487518479]
  We explore the influence of three key factors separately by transiting the modality from text to speech in an evolving manner. Factor A has a relatively minor impact, factor B influences syntactical and semantic modeling more obviously, and factor C exerts the most significant impact, particularly in the basic lexical modeling.
  arXiv  Detail & Related papers  (2024-12-22T14:59:19Z)
• The Impact of Token Granularity on the Predictive Power of Language Model Surprisal [15.073507986272027]
  One factor that has been overlooked in cognitive modeling is the granularity of subword tokens. Experiments with naturalistic reading times reveal a substantial influence of token granularity on surprisal. On garden-path constructions, language models trained on coarser-grained tokens generally assigned higher surprisal to critical regions.
  arXiv  Detail & Related papers  (2024-12-16T16:24:58Z)
• Counterfactual Generation from Language Models [64.55296662926919]
  We show that counterfactual reasoning is conceptually distinct from interventions. We propose a framework for generating true string counterfactuals. Our experiments demonstrate that the approach produces meaningful counterfactuals.
  arXiv  Detail & Related papers  (2024-11-11T17:57:30Z)
• Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models [16.328341121232484]
  We apply causal effect estimation strategies to measure the effect of context interventions. We investigate robustness to irrelevant changes and sensitivity to impactful changes of Transformers.
  arXiv  Detail & Related papers  (2024-04-03T10:22:35Z)
• The Hydra Effect: Emergent Self-repair in Language Model Computations [8.323441767835257]
  We investigate the internal structure of language model computations using causal analysis. We show two motifs: (1) a form of adaptive computation where ablations of one attention layer of a language model cause another layer to cause another layer. We analyse these effects in the context of factual recall and consider their implications for circuit-level attribution in language models.
  arXiv  Detail & Related papers  (2023-07-28T19:13:26Z)
• Traceability and Reuse Mechanisms, the most important Properties of Model Transformation Languages [1.4685355149711299]
  We aim to quantitatively asses the interview results to confirm or reject the effects posed by different factors. Results show that the Tracing and Reuse Mechanisms are most important overall.
  arXiv  Detail & Related papers  (2023-05-11T12:35:03Z)
• CausalDialogue: Modeling Utterance-level Causality in Conversations [83.03604651485327]
  We have compiled and expanded upon a new dataset called CausalDialogue through crowd-sourcing. This dataset includes multiple cause-effect pairs within a directed acyclic graph (DAG) structure. We propose a causality-enhanced method called Exponential Average Treatment Effect (ExMATE) to enhance the impact of causality at the utterance level in training neural conversation models.
  arXiv  Detail & Related papers  (2022-12-20T18:31:50Z)
• Discourse Context Predictability Effects in Hindi Word Order [14.88833412862455]
  We investigate how the words and syntactic structures in a sentence influence the word order of the following sentences. We use a number of discourse-based features and cognitive features to make its predictions, including dependency length, surprisal, and information status. We find that information status and LSTM-based discourse predictability influence word order choices, especially for non-canonical object-fronted orders.
  arXiv  Detail & Related papers  (2022-10-25T11:53:01Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Measuring the Impact of Individual Domain Factors in Self-Supervised Pre-Training [60.825471653739555]
  We show that phonetic domain factors play an important role during pre-training while grammatical and syntactic factors are far less important. This is the first study to better understand the domain characteristics of pre-trained sets in self-supervised pre-training for speech.
  arXiv  Detail & Related papers  (2022-03-01T17:40:51Z)
• Systematic Evaluation of Causal Discovery in Visual Model Based Reinforcement Learning [76.00395335702572]
  A central goal for AI and causality is the joint discovery of abstract representations and causal structure. Existing environments for studying causal induction are poorly suited for this objective because they have complicated task-specific causal graphs. In this work, our goal is to facilitate research in learning representations of high-level variables as well as causal structures among them.
  arXiv  Detail & Related papers  (2021-07-02T05:44:56Z)
• Do Language Embeddings Capture Scales? [54.1633257459927]
  We show that pretrained language models capture a significant amount of information about the scalar magnitudes of objects. We identify contextual information in pre-training and numeracy as two key factors affecting their performance.
  arXiv  Detail & Related papers  (2020-10-11T21:11:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.