Related papers: Compositional Causal Reasoning Evaluation in Language Models

Compositional Causal Reasoning Evaluation in Language Models

URL: http://arxiv.org/abs/2503.04556v2
Date: Sun, 16 Mar 2025 16:22:47 GMT
Title: Compositional Causal Reasoning Evaluation in Language Models
Authors: Jacqueline R. M. A. Maasch, Alihan Hüyük, Xinnuo Xu, Aditya V. Nori, Javier Gonzalez,
Abstract summary: Causal reasoning and compositional reasoning are two core aspirations in generative AI.<n>We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning ( CCR)<n>We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency.
Score: 18.138276908217023
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: Causal reasoning and compositional reasoning are two core aspirations in generative AI. Measuring the extent of these behaviors requires principled evaluation methods. We explore a unified perspective that considers both behaviors simultaneously, termed compositional causal reasoning (CCR): the ability to infer how causal measures compose and, equivalently, how causal quantities propagate through graphs. We instantiate a framework for the systematic evaluation of CCR for the average treatment effect and the probability of necessity and sufficiency. As proof of concept, we demonstrate the design of CCR tasks for language models in the LLama, Phi, and GPT families. On a math word problem, our framework revealed a range of taxonomically distinct error patterns. Additionally, CCR errors increased with the complexity of causal paths for all models except o1.

Related papers

Effective Bayesian Causal Inference via Structural Marginalisation and Autoregressive Orders [16.682775063684907]
We decompose the structure learning problem into inferring causal order and a parent set for each variable given a causal order. Our method yields state-of-the-art in structure learning on simulated non-linear additive noise benchmarks with scale-free and Erdos-Renyi graph structures.
arXiv Detail & Related papers (2024-02-22T18:39:24Z)
Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
arXiv Detail & Related papers (2023-08-24T16:06:36Z)
Causal models in string diagrams [0.0]
The framework of causal models provides a principled approach to causal reasoning, applied today across many scientific domains. We present this framework in the language of string diagrams, interpreted formally using category theory. We argue and demonstrate that causal reasoning according to the causal model framework is most naturally and intuitively done as diagrammatic reasoning.
arXiv Detail & Related papers (2023-04-15T21:54:48Z)
Neural Causal Models for Counterfactual Identification and Estimation [62.30444687707919]
We study the evaluation of counterfactual statements through neural models. First, we show that neural causal models (NCMs) are expressive enough. Second, we develop an algorithm for simultaneously identifying and estimating counterfactual distributions.
arXiv Detail & Related papers (2022-09-30T18:29:09Z)
Markov categories, causal theories, and the do-calculus [7.061298918159947]
We give a category-theoretic treatment of causal models that formalizes the syntax for causal reasoning over a directed acyclic graph (DAG) This framework enables us to define and study important concepts in causal reasoning from an abstract and "purely causal" point of view.
arXiv Detail & Related papers (2022-04-11T01:27:41Z)
Effect Identification in Cluster Causal Diagrams [51.42809552422494]
We introduce a new type of graphical model called cluster causal diagrams (for short, C-DAGs) C-DAGs allow for the partial specification of relationships among variables based on limited prior knowledge. We develop the foundations and machinery for valid causal inferences over C-DAGs.
arXiv Detail & Related papers (2022-02-22T21:27:31Z)
Causal Inference Principles for Reasoning about Commonsense Causality [93.19149325083968]
Commonsense causality reasoning aims at identifying plausible causes and effects in natural language descriptions that are deemed reasonable by an average person. Existing work usually relies on deep language models wholeheartedly, and is potentially susceptible to confounding co-occurrences. Motivated by classical causal principles, we articulate the central question of CCR and draw parallels between human subjects in observational studies and natural languages. We propose a novel framework, ROCK, to Reason O(A)bout Commonsense K(C)ausality, which utilizes temporal signals as incidental supervision.
arXiv Detail & Related papers (2022-01-31T06:12:39Z)
Learning Causal Semantic Representation for Out-of-Distribution Prediction [125.38836464226092]
We propose a Causal Semantic Generative model (CSG) based on a causal reasoning so that the two factors are modeled separately. We show that CSG can identify the semantic factor by fitting training data, and this semantic-identification guarantees the boundedness of OOD generalization error.
arXiv Detail & Related papers (2020-11-03T13:16:05Z)
A Critical View of the Structural Causal Model [89.43277111586258]
We show that one can identify the cause and the effect without considering their interaction at all. We propose a new adversarial training method that mimics the disentangled structure of the causal model. Our multidimensional method outperforms the literature methods on both synthetic and real world datasets.
arXiv Detail & Related papers (2020-02-23T22:52:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.