Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models

• URL: http://arxiv.org/abs/2505.19511v1
• Date: Mon, 26 May 2025 04:50:42 GMT
• Title: Causal Distillation: Transferring Structured Explanations from Large to Compact Language Models
• Authors: Aggrey Muhebwa, Khalid K. Osman,
• Abstract summary: Large proprietary language models exhibit strong causal reasoning abilities that smaller open-source models struggle to replicate.<n>We introduce a novel framework for distilling causal explanations that transfers causal reasoning skills from a powerful teacher model to a compact open-source model.<n>The key idea is to train the smaller model to develop causal reasoning abilities by generating structured cause-and-effect explanations consistent with those of the teacher model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large proprietary language models exhibit strong causal reasoning abilities that smaller open-source models struggle to replicate. We introduce a novel framework for distilling causal explanations that transfers causal reasoning skills from a powerful teacher model to a compact open-source model. The key idea is to train the smaller model to develop causal reasoning abilities by generating structured cause-and-effect explanations consistent with those of the teacher model. To evaluate the quality of the student-generated explanations, we introduce a new metric called Causal Explanation Coherence (CEC) to assess the structural and logical consistency of causal reasoning. This metric uses sentence-level semantic alignment to measure how well each part of the generated explanation corresponds to the teacher's reference, capturing both faithfulness and coverage of the underlying causal chain. Our framework and the CEC metric provide a principled foundation for training smaller models to perform robust causal reasoning and for systematically assessing the coherence of explanations in language model outputs.

Related papers

• Causal Reflection with Language Models [0.276240219662896]
  We introduce Causal Reflection, a framework that explicitly models causality as a dynamic function over state, action, time, and perturbation.<n>We define a formal Reflect mechanism that identifies mismatches between predicted and observed outcomes and generates causal hypotheses to revise the agent's internal model.<n>Our framework lays the theoretical groundwork for Causal Reflective agents that can adapt, self-correct, and communicate causal understanding in evolving environments.
  arXiv  Detail & Related papers  (2025-08-06T14:44:23Z)
• Failure Modes of LLMs for Causal Reasoning on Narratives [51.19592551510628]
  We investigate the interaction between world knowledge and logical reasoning.<n>We find that state-of-the-art large language models (LLMs) often rely on superficial generalizations.<n>We show that simple reformulations of the task can elicit more robust reasoning behavior.
  arXiv  Detail & Related papers  (2024-10-31T12:48:58Z)
• Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers [49.80959223722325]
  We study the distinction between feed-forward and attention layers in large language models.<n>We find that feed-forward layers tend to learn simple distributional associations such as bigrams, while attention layers focus on in-context reasoning.
  arXiv  Detail & Related papers  (2024-06-05T08:51:08Z)
• The Buffer Mechanism for Multi-Step Information Reasoning in Language Models [52.77133661679439]
  Investigating internal reasoning mechanisms of large language models can help us design better model architectures and training strategies. In this study, we constructed a symbolic dataset to investigate the mechanisms by which Transformer models employ vertical thinking strategy. We proposed a random matrix-based algorithm to enhance the model's reasoning ability, resulting in a 75% reduction in the training time required for the GPT-2 model.
  arXiv  Detail & Related papers  (2024-05-24T07:41:26Z)
• Cause and Effect: Can Large Language Models Truly Understand Causality? [1.2334534968968969]
  This research proposes a novel architecture called Context Aware Reasoning Enhancement with Counterfactual Analysis(CARE CA) framework. The proposed framework incorporates an explicit causal detection module with ConceptNet and counterfactual statements, as well as implicit causal detection through Large Language Models. The knowledge from ConceptNet enhances the performance of multiple causal reasoning tasks such as causal discovery, causal identification and counterfactual reasoning.
  arXiv  Detail & Related papers  (2024-02-28T08:02:14Z)
• Explainability for Large Language Models: A Survey [59.67574757137078]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. This paper introduces a taxonomy of explainability techniques and provides a structured overview of methods for explaining Transformer-based language models.
  arXiv  Detail & Related papers  (2023-09-02T22:14:26Z)
• Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
  We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
  arXiv  Detail & Related papers  (2023-08-24T16:06:36Z)
• MetaLogic: Logical Reasoning Explanations with Fine-Grained Structure [129.8481568648651]
  We propose a benchmark to investigate models' logical reasoning capabilities in complex real-life scenarios. Based on the multi-hop chain of reasoning, the explanation form includes three main components. We evaluate the current best models' performance on this new explanation form.
  arXiv  Detail & Related papers  (2022-10-22T16:01:13Z)
• Explaining Causal Models with Argumentation: the Case of Bi-variate Reinforcement [15.947501347927687]
  We introduce a conceptualisation for generating argumentation frameworks (AFs) from causal models. The conceptualisation is based on reinterpreting desirable properties of semantics of AFs as explanation moulds. We perform a theoretical evaluation of these argumentative explanations, examining whether they satisfy a range of desirable explanatory and argumentative properties.
  arXiv  Detail & Related papers  (2022-05-23T19:39:51Z)
• Thinking About Causation: A Causal Language with Epistemic Operators [58.720142291102135]
  We extend the notion of a causal model with a representation of the state of an agent. On the side of the object language, we add operators to express knowledge and the act of observing new information. We provide a sound and complete axiomatization of the logic, and discuss the relation of this framework to causal team semantics.
  arXiv  Detail & Related papers  (2020-10-30T12:16:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.