Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models

• URL: http://arxiv.org/abs/2604.22166v1
• Date: Fri, 24 Apr 2026 02:29:37 GMT
• Title: Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models
• Abstract summary: This study investigates whether models employ shared neural mechanisms across different syntactic constructions by applying causal interpretability methods at a granular level.<n>Our results reveal a highly localized and shared mechanism for filler-gap dependencies located in the early to middle layers.<n>We find that these mechanisms identified by activation patching generalize to out-of-distribution, while distributed alignment search, a supervised interpretability method, is susceptible to overfitting on narrow linguistic distributions.
• Score: 19.641608225642248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While language models demonstrate sophisticated syntactic capabilities, the extent to which their internal mechanisms align with cross-constructional principles studied in linguistics remains poorly understood. This study investigates whether models employ shared neural mechanisms across different syntactic constructions by applying causal interpretability methods at a granular level. Focusing on filler-gap dependencies and negative polarity item (NPI) licensing, we utilize activation patching to identify the functional roles of specific attention heads and MLP blocks. Our results reveal a highly localized and shared mechanism for filler-gap dependencies located in the early to middle layers, whereas NPI processing exhibits no such unified mechanism. Furthermore, we find that these mechanisms identified by activation patching generalize to out-of-distribution, while distributed alignment search, a supervised interpretability method, is susceptible to overfitting on narrow linguistic distributions. Finally, we validate our findings by demonstrating that the manipulation of the identified components improves model performance on acceptability judgment benchmarks.

Related papers

• Unleashing Multimodal Large Language Models for Training-free HOI Detection in the Wild [49.77540427384148]
  We present AgentHOI, a training-free, agentic framework that transfers the generalist multimodal reasoning capabilities of foundation models to HOI detection in the wild.<n>To address the challenges of incomplete interaction discovery and ambiguous localization in complex scenes, we introduce two key mechanisms.<n>AgentHOI achieves superior performance over state-of-the-art supervised and weakly supervised methods in real-world settings.
  arXiv  Detail & Related papers  (2026-07-15T14:30:20Z)
• K-Models: a Flexible and Interpretable Method for Ordinal Clustering with Application to Antigen-Antibody Interaction Profiles [65.21921642886367]
  K-Models is a novel framework that integrates ordinal constraints and estimates key underlying elements of the random process generating the observed functional profiles.<n>The proposed method is evaluated through simulations and real-world applications.
  arXiv  Detail & Related papers  (2026-05-14T13:35:44Z)
• Partially Observed Structural Causal Models [4.065430726803998]
  We introduce Partially Observed Structural Causal Models (POSCMs)<n>POSCMs formalize causal systems where latent contexts co-determine both the interaction structure and downstream mechanisms on observed variables.<n>We provide an identifiability theory that clarifies which intervention families would suffice to disentangle structure formation from mechanisms.
  arXiv  Detail & Related papers  (2026-05-05T01:40:14Z)
• Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models [77.98801218316505]
  Large language models (LLMs) exhibit emergent behaviors suggestive of human-like reasoning.<n>We investigate the internal processing of LLMs during in-context concept inference.
  arXiv  Detail & Related papers  (2026-02-08T03:14:39Z)
• DIML: Differentiable Inverse Mechanism Learning from Behaviors of Multi-Agent Learning Trajectories [7.764532811300023]
  We study inverse mechanism learning: recovering an unknown incentive-generating mechanism from observed strategic interaction traces.<n>Unlike inverse game theory and multi-agent inverse reinforcement learning, our target includes unstructured mechanism.<n>We propose DIML, a likelihood-based framework that differentiates through a model of multi-agent learning dynamics.
  arXiv  Detail & Related papers  (2026-01-25T03:49:25Z)
• State-Dependent Refusal and Learned Incapacity in RLHF-Aligned Language Models [0.0]
  We present a case-study methodology for auditing policy-linked behavioral selectivity in long-horizon interaction.<n>In a single 86-turn dialogue session, the same model shows Normal Performance (NP) in broad, non-sensitive domains while repeatedly producing Functional Refusal (FR) in provider- or policy-sensitive domains.<n>We operationalize three response regimes (NP, FR, Meta-Narrative; MN) and show that MN role-framing narratives tend to co-occur with refusals in the same sensitive contexts.
  arXiv  Detail & Related papers  (2025-12-15T14:00:15Z)
• Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations [50.45261187796993]
  Graph Neural Networks (GNNs) fail to fully utilize structural information, whereas Multi-Layer Perceptrons (MLPs) exhibit a surprising ability in structure-aware tasks.<n>This paper introduces a comprehensive probing framework from an information-theoretic perspective.
  arXiv  Detail & Related papers  (2025-06-26T18:10:28Z)
• Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations [31.54093034816165]
  Large Language Models (LLMs) exhibit a robust mastery of syntax when processing and generating text.<n>No comprehensive study has yet established whether a model's probing accuracy reliably predicts its downstream syntactic performance.
  arXiv  Detail & Related papers  (2025-06-20T01:46:50Z)
• Mechanistic Understanding and Mitigation of Language Confusion in English-Centric Large Language Models [56.61984030508691]
  We present the first mechanistic interpretability study of language confusion.<n>We show that confusion points (CPs) are central to this phenomenon.<n>We show that editing a small set of critical neurons, identified via comparative analysis with a multilingual-tuned counterpart, substantially mitigates confusion.
  arXiv  Detail & Related papers  (2025-05-22T11:29:17Z)
• Sensitivity Meets Sparsity: The Impact of Extremely Sparse Parameter Patterns on Theory-of-Mind of Large Language Models [55.46269953415811]
  We identify ToM-sensitive parameters and show that perturbing as little as 0.001% of these parameters significantly degrades ToM performance.<n>Our results have implications for enhancing model alignment, mitigating biases, and improving AI systems designed for human interaction.
  arXiv  Detail & Related papers  (2025-04-05T17:45:42Z)
• Reasoning Circuits in Language Models: A Mechanistic Interpretation of Syllogistic Inference [13.59675117792588]
  Recent studies on language models (LMs) have sparked a debate on whether they can learn systematic inferential principles.<n>This paper presents a mechanistic interpretation of syllogistic inference.
  arXiv  Detail & Related papers  (2024-08-16T07:47:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.