Discursive Circuits: How Do Language Models Understand Discourse Relations?

• URL: http://arxiv.org/abs/2510.11210v1
• Date: Mon, 13 Oct 2025 09:45:49 GMT
• Title: Discursive Circuits: How Do Language Models Understand Discourse Relations?
• Abstract summary: We show that sparse circuits generalize well to unseen discourse frameworks such as RST and SDRT.<n>Lower layers capture linguistic features such as lexical semantics and coreference, while upper layers encode discourse-level abstractions.
• Score: 24.164334889262175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Which components in transformer language models are responsible for discourse understanding? We hypothesize that sparse computational graphs, termed as discursive circuits, control how models process discourse relations. Unlike simpler tasks, discourse relations involve longer spans and complex reasoning. To make circuit discovery feasible, we introduce a task called Completion under Discourse Relation (CuDR), where a model completes a discourse given a specified relation. To support this task, we construct a corpus of minimal contrastive pairs tailored for activation patching in circuit discovery. Experiments show that sparse circuits ($\approx 0.2\%$ of a full GPT-2 model) recover discourse understanding in the English PDTB-based CuDR task. These circuits generalize well to unseen discourse frameworks such as RST and SDRT. Further analysis shows lower layers capture linguistic features such as lexical semantics and coreference, while upper layers encode discourse-level abstractions. Feature utility is consistent across frameworks (e.g., coreference supports Expansion-like relations).

Related papers

• Chained Recursive Language Models for Multi-Iteration Reasoning [46.765013720309064]
  Chained Recursive Language Models (Chained RLM) is an inference-time architecture.<n>Each root receives the original problem and context, but does not inherit the full conversational history.<n>In each staged computation, intermediate artifacts can be inspected, corrected, and extended by a later fresh inference by the same model.
  arXiv  Detail & Related papers  (2026-08-05T17:50:08Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• BeDiscovER: The Benchmark of Discourse Understanding in the Era of Reasoning Language Models [13.300475053766862]
  We introduce BeDiscovER, an up-to-date, comprehensive suite for evaluating the discourse-level knowledge of modern LLMs.<n>BeDiscovER compiles 5 publicly available discourse tasks across discourse lexicon, (multi-)sentential, and documental levels, with in total 52 individual datasets.
  arXiv  Detail & Related papers  (2025-11-17T07:50:12Z)
• Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering [51.7493726399073]
  We present a discourse-aware hierarchical framework to enhance long document question answering.<n>The framework involves three key innovations: specialized discourse parsing for lengthy documents, LLM-based enhancement of discourse relation nodes, and structure-guided hierarchical retrieval.
  arXiv  Detail & Related papers  (2025-05-26T14:45:12Z)
• Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models [55.19497659895122]
  We introduce methods for discovering and applying sparse feature circuits.<n>These are causally implicatedworks of human-interpretable features for explaining language model behaviors.
  arXiv  Detail & Related papers  (2024-03-28T17:56:07Z)
• Transformer-Based Models Are Not Yet Perfect At Learning to Emulate Structural Recursion [14.739369424331478]
  We introduce a general framework that nicely connects the abstract concepts of structural recursion in the programming language domain to sequence modeling problems and learned models' behavior. With our framework as a powerful conceptual tool, we identify different issues under various set-ups.
  arXiv  Detail & Related papers  (2024-01-23T18:07:38Z)
• Towards Interpretable Sequence Continuation: Analyzing Shared Circuits in Large Language Models [9.56229382432426]
  This research aims to reverse engineer transformer models into human-readable representations that implement algorithmic functions. By applying circuit interpretability analysis, we identify a key sub-circuit in both GPT-2 Small and Llama-2-7B. We show that this sub-circuit has effects on various math-related prompts, such as on intervaled circuits, Spanish number word and months continuation, and natural language word problems.
  arXiv  Detail & Related papers  (2023-11-07T16:58:51Z)
• Solving Hard Analogy Questions with Relation Embedding Chains [21.31821067426995]
  Modelling how concepts are related is a central topic in Lexical Semantics. KGs are limited to a fixed set of relation types, and they are incomplete and often noisy. We distill relation embeddings from a fine-tuned language model. We show that our proposed representations are useful for solving hard analogy questions.
  arXiv  Detail & Related papers  (2023-10-18T23:13:22Z)
• Modeling Hierarchical Reasoning Chains by Linking Discourse Units and Key Phrases for Reading Comprehension [80.99865844249106]
  We propose a holistic graph network (HGN) which deals with context at both discourse level and word level, as the basis for logical reasoning. Specifically, node-level and type-level relations, which can be interpreted as bridges in the reasoning process, are modeled by a hierarchical interaction mechanism.
  arXiv  Detail & Related papers  (2023-06-21T07:34:27Z)
• DiscoPrompt: Path Prediction Prompt Tuning for Implicit Discourse Relation Recognition [27.977742959064916]
  We propose a prompt-based path prediction method to utilize the interactive information and intrinsic senses among the hierarchy in IDRR. This is the first work that injects such structure information into pre-trained language models via prompt tuning.
  arXiv  Detail & Related papers  (2023-05-06T08:16:07Z)
• RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction [65.4337085607711]
  We introduce the task setting of Zero-Shot Relation Triplet Extraction (ZeroRTE) Given an input sentence, each extracted triplet consists of the head entity, relation label, and tail entity where the relation label is not seen at the training stage. We propose to synthesize relation examples by prompting language models to generate structured texts.
  arXiv  Detail & Related papers  (2022-03-17T05:55:14Z)
• Probing Linguistic Features of Sentence-Level Representations in Neural Relation Extraction [80.38130122127882]
  We introduce 14 probing tasks targeting linguistic properties relevant to neural relation extraction (RE) We use them to study representations learned by more than 40 different encoder architecture and linguistic feature combinations trained on two datasets. We find that the bias induced by the architecture and the inclusion of linguistic features are clearly expressed in the probing task performance.
  arXiv  Detail & Related papers  (2020-04-17T09:17:40Z)
• Multi-Step Inference for Reasoning Over Paragraphs [95.91527524872832]
  Complex reasoning over text requires understanding and chaining together free-form predicates and logical connectives. We present a compositional model reminiscent of neural module networks that can perform chained logical reasoning.
  arXiv  Detail & Related papers  (2020-04-06T21:12:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.