Thought Branches: Interpreting LLM Reasoning Requires Resampling

• URL: http://arxiv.org/abs/2510.27484v1
• Date: Fri, 31 Oct 2025 14:02:37 GMT
• Title: Thought Branches: Interpreting LLM Reasoning Requires Resampling
• Abstract summary: We argue that studying a single sample is inadequate for understanding causal influence and the underlying computation.<n>We present case studies using resampling to investigate model decisions.
• Score: 10.978878422025232
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most work interpreting reasoning models studies only a single chain-of-thought (CoT), yet these models define distributions over many possible CoTs. We argue that studying a single sample is inadequate for understanding causal influence and the underlying computation. Though fully specifying this distribution is intractable, it can be understood by sampling. We present case studies using resampling to investigate model decisions. First, when a model states a reason for its action, does that reason actually cause the action? In "agentic misalignment" scenarios, we resample specific sentences to measure their downstream effects. Self-preservation sentences have small causal impact, suggesting they do not meaningfully drive blackmail. Second, are artificial edits to CoT sufficient for steering reasoning? These are common in literature, yet take the model off-policy. Resampling and selecting a completion with the desired property is a principled on-policy alternative. We find off-policy interventions yield small and unstable effects compared to resampling in decision-making tasks. Third, how do we understand the effect of removing a reasoning step when the model may repeat it post-edit? We introduce a resilience metric that repeatedly resamples to prevent similar content from reappearing downstream. Critical planning statements resist removal but have large effects when eliminated. Fourth, since CoT is sometimes "unfaithful", can our methods teach us anything in these settings? Adapting causal mediation analysis, we find that hints that have a causal effect on the output without being explicitly mentioned exert a subtle and cumulative influence on the CoT that persists even if the hint is removed. Overall, studying distributions via resampling enables reliable causal analysis, clearer narratives of model reasoning, and principled CoT interventions.

Related papers

• Pando: Do Interpretability Methods Work When Models Won't Explain Themselves? [53.07826484214082]
  We introduce Pando, a model-organism benchmark that breaks this confound via an explanation axis.<n>Pando predicts held-out model decisions from $10$ labeled query-response pairs.<n>When explanations are faithful, black-box elicitation matches or exceeds all white-box methods.
  arXiv  Detail & Related papers  (2026-04-13T06:42:24Z)
• Reasoning Traces Shape Outputs but Models Won't Say So [11.928107300132062]
  We introduce Thought Injection, a method that injects synthetic reasoning snippets into a model's think> trace.<n>We find that injected hints reliably alter outputs, confirming that reasoning traces causally shape model behavior.<n>When asked to explain their changed answers, models overwhelmingly refuse to disclose the influence.
  arXiv  Detail & Related papers  (2026-03-21T03:31:36Z)
• Decoding Answers Before Chain-of-Thought: Evidence from Pre-CoT Probes and Activation Steering [5.427346259545067]
  Chain-of-thought (CoT) has become central to scaling reasoning capabilities in large language models.<n>We show that instruction-tuned models often determine their answer before generating CoT.
  arXiv  Detail & Related papers  (2026-03-02T04:33:55Z)
• Mitigating Spurious Correlations Between Question and Answer via Chain-of-Thought Correctness Perception Distillation [25.195244084313114]
  Chain-of-Thought Correctness Perception Distillation (CoPeD) aims to improve the reasoning quality of the student model.<n>CoPeD encourages the student model to predict answers based on correct rationales and revise them when they are incorrect.
  arXiv  Detail & Related papers  (2025-09-06T05:33:17Z)
• Boosting Temporal Sentence Grounding via Causal Inference [55.61521060331558]
  Temporal Sentence Grounding aims to identify relevant moments in an untrimmed video that semantically correspond to a given textual query.<n>These spurious correlations arise from two primary factors: (1) inherent biases in the textual data, such as frequent co-occurrences of specific verbs or phrases, and (2) the model's tendency to overfit to salient or repetitive patterns in video content.<n>We propose a novel TSG framework, causal intervention and counterfactual reasoning that utilize causal inference to eliminate spurious correlations and enhance the model's robustness.
  arXiv  Detail & Related papers  (2025-07-07T13:01:06Z)
• Counterfactual reasoning: an analysis of in-context emergence [57.118735341305786]
  We show that language models are capable of counterfactual reasoning.<n>We find that self-attention, model depth and pre-training data diversity drive performance.<n>Our findings extend to counterfactual reasoning under SDE dynamics.
  arXiv  Detail & Related papers  (2025-06-05T16:02:07Z)
• Counterfactual Realizability [52.85109506684737]
  We introduce a formal definition of realizability, the ability to draw samples from a distribution, and then develop a complete algorithm to determine whether an arbitrary counterfactual distribution is realizable.<n>We illustrate the implications of this new framework for counterfactual data collection using motivating examples from causal fairness and causal reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-14T20:54:27Z)
• Rethinking harmless refusals when fine-tuning foundation models [0.8571111167616167]
  We investigate the degree to which fine-tuning in Large Language Models (LLMs) effectively mitigates versus merely conceals undesirable behavior. We identify a pervasive phenomenon we term emphreason-based deception, where models either stop producing reasoning traces or produce seemingly ethical reasoning traces that belie the unethical nature of their final outputs.
  arXiv  Detail & Related papers  (2024-06-27T22:08:22Z)
• Preemptive Answer "Attacks" on Chain-of-Thought Reasoning [7.233752893356647]
  Large language models (LLMs) showcase impressive reasoning capabilities when coupled with Chain-of-Thought prompting. In this paper, we introduce a novel scenario termed preemptive answers, where the LLM obtains an answer before engaging in reasoning. Experiments reveal that preemptive answers significantly impair the model's reasoning capability across various CoT methods and a broad spectrum of datasets.
  arXiv  Detail & Related papers  (2024-05-31T15:15:04Z)
• Navigating the OverKill in Large Language Models [84.62340510027042]
  We investigate the factors for overkill by exploring how models handle and determine the safety of queries. Our findings reveal the presence of shortcuts within models, leading to an over-attention of harmful words like 'kill' and prompts emphasizing safety will exacerbate overkill. We introduce Self-Contrastive Decoding (Self-CD), a training-free and model-agnostic strategy, to alleviate this phenomenon.
  arXiv  Detail & Related papers  (2024-01-31T07:26:47Z)
• Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting [43.458726163197824]
  Large Language Models (LLMs) can achieve strong performance on many tasks by producing step-by-step reasoning before giving a final output. We find that CoT explanations can systematically misrepresent the true reason for a model's prediction.
  arXiv  Detail & Related papers  (2023-05-07T22:44:25Z)
• Active Bayesian Causal Inference [72.70593653185078]
  We propose Active Bayesian Causal Inference (ABCI), a fully-Bayesian active learning framework for integrated causal discovery and reasoning. ABCI jointly infers a posterior over causal models and queries of interest. We show that our approach is more data-efficient than several baselines that only focus on learning the full causal graph.
  arXiv  Detail & Related papers  (2022-06-04T22:38:57Z)
• Causal Expectation-Maximisation [70.45873402967297]
  We show that causal inference is NP-hard even in models characterised by polytree-shaped graphs. We introduce the causal EM algorithm to reconstruct the uncertainty about the latent variables from data about categorical manifest variables. We argue that there appears to be an unnoticed limitation to the trending idea that counterfactual bounds can often be computed without knowledge of the structural equations.
  arXiv  Detail & Related papers  (2020-11-04T10:25:13Z)
• Superdeterministic hidden-variables models I: nonequilibrium and signalling [0.0]
  We first give an overview of superdeterminism and discuss various criticisms of it raised in the literature. We take up Bell's intuitive criticism that these models are conspiratorial'
  arXiv  Detail & Related papers  (2020-03-26T15:49:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.