A Design-based Solution for Causal Inference with Text: Can a Language Model Be Too Large?

• URL: http://arxiv.org/abs/2510.08758v1
• Date: Thu, 09 Oct 2025 19:17:57 GMT
• Title: A Design-based Solution for Causal Inference with Text: Can a Language Model Be Too Large?
• Authors: Graham Tierney, Srikar Katta, Christopher Bail, Sunshine Hillygus, Alexander Volfovsky,
• Abstract summary: Social science questions ask how linguistic properties causally affect an audience's attitudes and behaviors.<n>Recent literature proposes adapting large language models to learn latent representations of text that successfully predict both treatment and the outcome.<n>We introduce a new experimental design that handles latent confounding, avoids the overlap issue, and unbiasedly estimates treatment effects.
• Score: 39.56626912746574
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Many social science questions ask how linguistic properties causally affect an audience's attitudes and behaviors. Because text properties are often interlinked (e.g., angry reviews use profane language), we must control for possible latent confounding to isolate causal effects. Recent literature proposes adapting large language models (LLMs) to learn latent representations of text that successfully predict both treatment and the outcome. However, because the treatment is a component of the text, these deep learning methods risk learning representations that actually encode the treatment itself, inducing overlap bias. Rather than depending on post-hoc adjustments, we introduce a new experimental design that handles latent confounding, avoids the overlap issue, and unbiasedly estimates treatment effects. We apply this design in an experiment evaluating the persuasiveness of expressing humility in political communication. Methodologically, we demonstrate that LLM-based methods perform worse than even simple bag-of-words models using our real text and outcomes from our experiment. Substantively, we isolate the causal effect of expressing humility on the perceived persuasiveness of political statements, offering new insights on communication effects for social media platforms, policy makers, and social scientists.

Related papers

• Causal Inference on Outcomes Learned from Text [1.283555556182245]
  We propose a machine-learning tool that yields causal inference on text in randomized trials.<n>Based on a simple econometric framework, our procedure addresses three questions: First, is the text affected by the treatment? Second, which outcomes is the effect on? And third, how complete is our description of causal effects?
  arXiv  Detail & Related papers  (2025-03-02T04:36:27Z)
• Modulating Language Model Experiences through Frictions [56.17593192325438]
  Over-consumption of language model outputs risks propagating unchecked errors in the short-term and damaging human capabilities for critical thinking in the long-term. We propose selective frictions for language model experiences, inspired by behavioral science interventions, to dampen misuse.
  arXiv  Detail & Related papers  (2024-06-24T16:31:11Z)
• Outcome-Constrained Large Language Models for Countering Hate Speech [10.434435022492723]
  This study aims to develop methods for generating counterspeech constrained by conversation outcomes. We experiment with large language models (LLMs) to incorporate into the text generation process two desired conversation outcomes. Evaluation results show that our methods effectively steer the generation of counterspeech toward the desired outcomes.
  arXiv  Detail & Related papers  (2024-03-25T19:44:06Z)
• Natural Language Decompositions of Implicit Content Enable Better Text Representations [52.992875653864076]
  We introduce a method for the analysis of text that takes implicitly communicated content explicitly into account.<n>We use a large language model to produce sets of propositions that are inferentially related to the text that has been observed.<n>Our results suggest that modeling the meanings behind observed language, rather than the literal text alone, is a valuable direction for NLP.
  arXiv  Detail & Related papers  (2023-05-23T23:45:20Z)
• Do Large Language Models Show Decision Heuristics Similar to Humans? A Case Study Using GPT-3.5 [0.0]
  GPT-3.5 is an example of an LLM that supports a conversational agent called ChatGPT. In this work, we used a series of novel prompts to determine whether ChatGPT shows biases, and other decision effects. We also tested the same prompts on human participants.
  arXiv  Detail & Related papers  (2023-05-08T01:02:52Z)
• Language with Vision: a Study on Grounded Word and Sentence Embeddings [6.231247903840833]
  Grounding language in vision is an active field of research seeking to construct cognitively plausible word and sentence representations. The present study proposes a computational grounding model for pre-trained word embeddings. Our model effectively balances the interplay between language and vision by aligning textual embeddings with visual information.
  arXiv  Detail & Related papers  (2022-06-17T15:04:05Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• Causal Effects of Linguistic Properties [41.65859219291606]
  We consider the problem of using observational data to estimate the causal effects of linguistic properties. We introduce TextCause, an algorithm for estimating causal effects of linguistic properties. We show that the proposed method outperforms related approaches when estimating the effect of Amazon review sentiment.
  arXiv  Detail & Related papers  (2020-10-24T15:43:37Z)
• Probing Contextual Language Models for Common Ground with Visual Representations [76.05769268286038]
  We design a probing model that evaluates how effective are text-only representations in distinguishing between matching and non-matching visual representations. Our findings show that language representations alone provide a strong signal for retrieving image patches from the correct object categories. Visually grounded language models slightly outperform text-only language models in instance retrieval, but greatly under-perform humans.
  arXiv  Detail & Related papers  (2020-05-01T21:28:28Z)
• A Deep Neural Framework for Contextual Affect Detection [51.378225388679425]
  A short and simple text carrying no emotion can represent some strong emotions when reading along with its context. We propose a Contextual Affect Detection framework which learns the inter-dependence of words in a sentence.
  arXiv  Detail & Related papers  (2020-01-28T05:03:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.