Causal Inference on Outcomes Learned from Text

• URL: http://arxiv.org/abs/2503.00725v1
• Date: Sun, 02 Mar 2025 04:36:27 GMT
• Title: Causal Inference on Outcomes Learned from Text
• Authors: Iman Modarressi, Jann Spiess, Amar Venugopal,
• Abstract summary: We propose a machine-learning tool that yields causal inference on text in randomized trials.<n>Based on a simple econometric framework, our procedure addresses three questions: First, is the text affected by the treatment? Second, which outcomes is the effect on? And third, how complete is our description of causal effects?
• Score: 1.283555556182245
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a machine-learning tool that yields causal inference on text in randomized trials. Based on a simple econometric framework in which text may capture outcomes of interest, our procedure addresses three questions: First, is the text affected by the treatment? Second, which outcomes is the effect on? And third, how complete is our description of causal effects? To answer all three questions, our approach uses large language models (LLMs) that suggest systematic differences across two groups of text documents and then provides valid inference based on costly validation. Specifically, we highlight the need for sample splitting to allow for statistical validation of LLM outputs, as well as the need for human labeling to validate substantive claims about how documents differ across groups. We illustrate the tool in a proof-of-concept application using abstracts of academic manuscripts.

Related papers

• A Design-based Solution for Causal Inference with Text: Can a Language Model Be Too Large? [39.56626912746574]
  Social science questions ask how linguistic properties causally affect an audience's attitudes and behaviors.<n>Recent literature proposes adapting large language models to learn latent representations of text that successfully predict both treatment and the outcome.<n>We introduce a new experimental design that handles latent confounding, avoids the overlap issue, and unbiasedly estimates treatment effects.
  arXiv  Detail & Related papers  (2025-10-09T19:17:57Z)
• Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation [46.697788643450785]
  Large language models (LLMs) have been found to produce outputs that are incomplete or selectively omit key information.<n>In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies.
  arXiv  Detail & Related papers  (2025-10-09T08:22:24Z)
• RepreGuard: Detecting LLM-Generated Text by Revealing Hidden Representation Patterns [50.401907401444404]
  Large language models (LLMs) are crucial for preventing misuse and building trustworthy AI systems.<n>We propose RepreGuard, an efficient statistics-based detection method.<n> Experimental results show that RepreGuard outperforms all baselines with average 94.92% AUROC on both in-distribution (ID) and OOD scenarios.
  arXiv  Detail & Related papers  (2025-08-18T17:59:15Z)
• Verified Language Processing with Hybrid Explainability: A Technical Report [0.7066382982173529]
  We present a novel pipeline designed for hybrid explainability to address this.<n>Our methodology combines graphs and logic to produce First-Order Logic representations, creating machine- and human-readable representations through Montague Grammar.<n>Preliminary results indicate the effectiveness of this approach in capturing full text similarity.
  arXiv  Detail & Related papers  (2025-07-07T14:00:05Z)
• ExaGPT: Example-Based Machine-Generated Text Detection for Human Interpretability [62.285407189502216]
  Detecting texts generated by Large Language Models (LLMs) could cause grave mistakes due to incorrect decisions.<n>We introduce ExaGPT, an interpretable detection approach grounded in the human decision-making process.<n>We show that ExaGPT massively outperforms prior powerful detectors by up to +40.9 points of accuracy at a false positive rate of 1%.
  arXiv  Detail & Related papers  (2025-02-17T01:15:07Z)
• Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models [30.066436019078164]
  We study what kind of generalisation strategies Large Language Models employ when performing reasoning tasks. Our findings indicate that the approach to reasoning the models use is unlike retrieval, and more like a generalisable strategy.
  arXiv  Detail & Related papers  (2024-11-19T15:47:12Z)
• Interactive Analysis of LLMs using Meaningful Counterfactuals [22.755345889167934]
  Counterfactual examples are useful for exploring the decision boundaries of machine learning models. How can we apply counterfactual-based methods to analyze and explain LLMs? We propose a novel algorithm for generating batches of complete and meaningful textual counterfactuals. In our experiments, 97.2% of the counterfactuals are grammatically correct.
  arXiv  Detail & Related papers  (2024-04-23T19:57:03Z)
• Protecting Copyrighted Material with Unique Identifiers in Large Language Model Training [55.321010757641524]
  A primary concern regarding training large language models (LLMs) is whether they abuse copyrighted online text.<n>We propose an alternative textitinsert-and-detect methodology, advocating that web users and content platforms employ textbftextitunique identifiers for reliable and independent membership inference.
  arXiv  Detail & Related papers  (2024-03-23T06:36:32Z)
• Enhancing Argument Structure Extraction with Efficient Leverage of Contextual Information [79.06082391992545]
  We propose an Efficient Context-aware model (ECASE) that fully exploits contextual information. We introduce a sequence-attention module and distance-weighted similarity loss to aggregate contextual information and argumentative information. Our experiments on five datasets from various domains demonstrate that our model achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-10-08T08:47:10Z)
• ICL-D3IE: In-Context Learning with Diverse Demonstrations Updating for Document Information Extraction [56.790794611002106]
  Large language models (LLMs) have demonstrated remarkable results in various natural language processing (NLP) tasks with in-context learning. We propose a simple but effective in-context learning framework called ICL-D3IE. Specifically, we extract the most difficult and distinct segments from hard training documents as hard demonstrations.
  arXiv  Detail & Related papers  (2023-03-09T06:24:50Z)
• Revisiting text decomposition methods for NLI-based factuality scoring of summaries [9.044665059626958]
  We show that fine-grained decomposition is not always a winning strategy for factuality scoring. We also show that small changes to previously proposed entailment-based scoring methods can result in better performance.
  arXiv  Detail & Related papers  (2022-11-30T09:54:37Z)
• Complementary Explanations for Effective In-Context Learning [77.83124315634386]
  Large language models (LLMs) have exhibited remarkable capabilities in learning from explanations in prompts. This work aims to better understand the mechanisms by which explanations are used for in-context learning.
  arXiv  Detail & Related papers  (2022-11-25T04:40:47Z)
• Dense Paraphrasing for Textual Enrichment [7.6233489924270765]
  We define the process of rewriting a textual expression (lexeme or phrase) such that it reduces ambiguity while also making explicit the underlying semantics that is not (necessarily) expressed in the economy of sentence structure as Dense Paraphrasing (DP) We build the first complete DP dataset, provide the scope and design of the annotation task, and present results demonstrating how this DP process can enrich a source text to improve inferencing and QA task performance.
  arXiv  Detail & Related papers  (2022-10-20T19:58:31Z)
• Textual Entailment Recognition with Semantic Features from Empirical Text Representation [60.31047947815282]
  A text entails a hypothesis if and only if the true value of the hypothesis follows the text. In this paper, we propose a novel approach to identifying the textual entailment relationship between text and hypothesis. We employ an element-wise Manhattan distance vector-based feature that can identify the semantic entailment relationship between the text-hypothesis pair.
  arXiv  Detail & Related papers  (2022-10-18T10:03:51Z)
• GERE: Generative Evidence Retrieval for Fact Verification [57.78768817972026]
  We propose GERE, the first system that retrieves evidences in a generative fashion. The experimental results on the FEVER dataset show that GERE achieves significant improvements over the state-of-the-art baselines.
  arXiv  Detail & Related papers  (2022-04-12T03:49:35Z)
• Generating Fact Checking Summaries for Web Claims [8.980876474818153]
  We present a neural attention-based approach that learns to establish the correctness of textual claims based on evidence in the form of text documents. We show the efficacy of our approach on datasets concerning political, healthcare, and environmental issues.
  arXiv  Detail & Related papers  (2020-10-16T18:10:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.