Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs

• URL: http://arxiv.org/abs/2604.13950v1
• Date: Wed, 15 Apr 2026 15:03:33 GMT
• Title: Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs
• Abstract summary: We show how causal interventions in Transformer models provide insights into English syntax by focusing on a long-standing challenge for syntactic theory: syntactic islands.<n>We show that modern Transformer language models replicate human judgments across this gradient.<n>We derive a novel linguistic hypothesis: the conjunction "and" is represented differently in extractable versus non-extractable constructions.
• Score: 11.31435294855236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show how causal interventions in Transformer models provide insights into English syntax by focusing on a long-standing challenge for syntactic theory: syntactic islands. Extraction from coordinated verb phrases is often degraded, yet acceptability varies gradiently with lexical content (e.g., "I know what he hates art and loves" vs. "I know what he looked down and saw"). We show that modern Transformer language models replicate human judgments across this gradient. Using causal interventions that isolate functionally relevant subspaces in Transformer blocks, attention modules, and MLPs, we demonstrate that extraction from coordination islands engages the same filler-gap mechanisms as canonical wh-dependencies, but that these mechanisms are selectively blocked to varying degrees. By projecting a large corpus of unrelated text onto these causally identified subspaces, we derive a novel linguistic hypothesis: the conjunction "and" is represented differently in extractable versus non-extractable constructions, corresponding to expressions encoding relational dependencies versus purely conjunctive uses. These results illustrate how mechanistic interpretability can inform syntax, generating new hypotheses about linguistic representation and processing.

Related papers

• Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations [31.54093034816165]
  Large Language Models (LLMs) exhibit a robust mastery of syntax when processing and generating text.<n>No comprehensive study has yet established whether a model's probing accuracy reliably predicts its downstream syntactic performance.
  arXiv  Detail & Related papers  (2025-06-20T01:46:50Z)
• A Transformer and Prototype-based Interpretable Model for Contextual Sarcasm Detection [2.415131843893681]
  Sarcasm detection, with its figurative nature, poses unique challenges for affective systems designed to perform sentiment analysis.<n>We propose a method that leverages LMs and prototype-based networks, enhanced by sentiment embeddings to conduct interpretable sarcasm detection.<n>Our approach is intrinsically interpretable without extra post-hoc interpretability techniques.
  arXiv  Detail & Related papers  (2025-03-14T19:58:43Z)
• Explaining Text Similarity in Transformer Models [52.571158418102584]
  Recent advances in explainable AI have made it possible to mitigate limitations by leveraging improved explanations for Transformers. We use BiLRP, an extension developed for computing second-order explanations in bilinear similarity models, to investigate which feature interactions drive similarity in NLP models. Our findings contribute to a deeper understanding of different semantic similarity tasks and models, highlighting how novel explainable AI methods enable in-depth analyses and corpus-level insights.
  arXiv  Detail & Related papers  (2024-05-10T17:11:31Z)
• Probabilistic Transformer: A Probabilistic Dependency Model for Contextual Word Representation [52.270712965271656]
  We propose a new model of contextual word representation, not from a neural perspective, but from a purely syntactic and probabilistic perspective. We find that the graph of our model resembles transformers, with correspondences between dependencies and self-attention. Experiments show that our model performs competitively to transformers on small to medium sized datasets.
  arXiv  Detail & Related papers  (2023-11-26T06:56:02Z)
• Assessing the Capacity of Transformer to Abstract Syntactic Representations: A Contrastive Analysis Based on Long-distance Agreement [4.995343972237368]
  The long-distance agreement is increasingly used to assess the syntactic generalization of Neural Language Models. Much work has shown that transformers are capable of high accuracy in varied agreement tasks. This work contrasts how they handle two superficially similar but theoretically distinct agreement phenomena.
  arXiv  Detail & Related papers  (2022-12-08T19:10:46Z)
• Latent Topology Induction for Understanding Contextualized Representations [84.7918739062235]
  We study the representation space of contextualized embeddings and gain insight into the hidden topology of large language models. We show there exists a network of latent states that summarize linguistic properties of contextualized representations.
  arXiv  Detail & Related papers  (2022-06-03T11:22:48Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Unsupervised Distillation of Syntactic Information from Contextualized Word Representations [62.230491683411536]
  We tackle the task of unsupervised disentanglement between semantics and structure in neural language representations. To this end, we automatically generate groups of sentences which are structurally similar but semantically different. We demonstrate that our transformation clusters vectors in space by structural properties, rather than by lexical semantics.
  arXiv  Detail & Related papers  (2020-10-11T15:13:18Z)
• Discourse structure interacts with reference but not syntax in neural language models [17.995905582226463]
  We study the ability of language models (LMs) to learn interactions between different linguistic representations. We find that, contrary to humans, implicit causality only influences LM behavior for reference, not syntax. Our results suggest that LM behavior can contradict not only learned representations of discourse but also syntactic agreement.
  arXiv  Detail & Related papers  (2020-10-10T03:14:00Z)
• Constructing a Family Tree of Ten Indo-European Languages with Delexicalized Cross-linguistic Transfer Patterns [57.86480614673034]
  We formalize the delexicalized transfer as interpretable tree-to-string and tree-to-tree patterns. This allows us to quantitatively probe cross-linguistic transfer and extend inquiries of Second Language Acquisition.
  arXiv  Detail & Related papers  (2020-07-17T15:56:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.