Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference

• URL: http://arxiv.org/abs/2009.09961v4
• Date: Fri, 6 May 2022 06:33:22 GMT
• Title: Adjusting for Confounders with Text: Challenges and an Empirical Evaluation Framework for Causal Inference
• Authors: Galen Weld, Peter West, Maria Glenski, David Arbour, Ryan Rossi, Tim Althoff
• Abstract summary: Causal inference studies using textual social media data can provide actionable insights on human behavior. No empirical evaluation framework for causal methods using text exists. Our framework enables the evaluation of any casual inference method using text.
• Score: 15.506820260229256
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Causal inference studies using textual social media data can provide actionable insights on human behavior. Making accurate causal inferences with text requires controlling for confounding which could otherwise impart bias. Recently, many different methods for adjusting for confounders have been proposed, and we show that these existing methods disagree with one another on two datasets inspired by previous social media studies. Evaluating causal methods is challenging, as ground truth counterfactuals are almost never available. Presently, no empirical evaluation framework for causal methods using text exists, and as such, practitioners must select their methods without guidance. We contribute the first such framework, which consists of five tasks drawn from real world studies. Our framework enables the evaluation of any casual inference method using text. Across 648 experiments and two datasets, we evaluate every commonly used causal inference method and identify their strengths and weaknesses to inform social media researchers seeking to use such methods, and guide future improvements. We make all tasks, data, and models public to inform applications and encourage additional research.

Related papers

• Bias and Fairness in Large Language Models: A Survey [73.87651986156006]
  We present a comprehensive survey of bias evaluation and mitigation techniques for large language models (LLMs) We first consolidate, formalize, and expand notions of social bias and fairness in natural language processing. We then unify the literature by proposing three intuitive, two for bias evaluation, and one for mitigation.
  arXiv  Detail & Related papers  (2023-09-02T00:32:55Z)
• Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectives [11.63431725146897]
  The trustworthiness of machine learning has emerged as a critical topic in the field. This survey presents the background of trustworthy machine learning development using a unified set of concepts. We provide a unified language with mathematical vocabulary to link these methods across robustness, adversarial robustness, interpretability, and fairness.
  arXiv  Detail & Related papers  (2023-07-31T17:11:35Z)
• Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems [82.92678837778358]
  preference-based methods have demonstrated substantial success in empirical applications such as InstructGPT. We show how human bias and uncertainty in feedback modelings can affect the theoretical guarantees of these approaches.
  arXiv  Detail & Related papers  (2023-07-24T17:50:24Z)
• Integrity and Junkiness Failure Handling for Embedding-based Retrieval: A Case Study in Social Network Search [26.705196461992845]
  Embedding based retrieval has seen its usage in a variety of search applications like e-commerce, social networking search etc. In this paper, we conduct an analysis of embedding-based retrieval launched in early 2021 on our social network search engine. We define two main categories of failures introduced by it, integrity and junkiness.
  arXiv  Detail & Related papers  (2023-04-18T20:53:47Z)
• Robust and Explainable Identification of Logical Fallacies in Natural Language Arguments [5.850977561881791]
  We formalize prior theoretical work on logical fallacies into a comprehensive three-stage evaluation framework. We employ three families of robust and explainable methods based on prototype reasoning, instance-based reasoning, and knowledge injection. We extensively evaluate these methods on our datasets, focusing on their robustness and explainability.
  arXiv  Detail & Related papers  (2022-12-12T20:27:17Z)
• Empirical Estimates on Hand Manipulation are Recoverable: A Step Towards Individualized and Explainable Robotic Support in Everyday Activities [80.37857025201036]
  Key challenge for robotic systems is to figure out the behavior of another agent. Processing correct inferences is especially challenging when (confounding) factors are not controlled experimentally. We propose equipping robots with the necessary tools to conduct observational studies on people.
  arXiv  Detail & Related papers  (2022-01-27T22:15:56Z)
• Individual Explanations in Machine Learning Models: A Case Study on Poverty Estimation [63.18666008322476]
  Machine learning methods are being increasingly applied in sensitive societal contexts. The present case study has two main objectives. First, to expose these challenges and how they affect the use of relevant and novel explanations methods. And second, to present a set of strategies that mitigate such challenges, as faced when implementing explanation methods in a relevant application domain.
  arXiv  Detail & Related papers  (2021-04-09T01:54:58Z)
• A Survey on Text Classification: From Shallow to Deep Learning [83.47804123133719]
  The last decade has seen a surge of research in this area due to the unprecedented success of deep learning. This paper fills the gap by reviewing the state-of-the-art approaches from 1961 to 2021. We create a taxonomy for text classification according to the text involved and the models used for feature extraction and classification.
  arXiv  Detail & Related papers  (2020-08-02T00:09:03Z)
• Text and Causal Inference: A Review of Using Text to Remove Confounding from Causal Estimates [15.69581581445705]
  An individual's entire history of social media posts or the content of a news article could provide a rich measurement of confounders. Despite increased attention on adjusting for confounding using text, there are still many open problems.
  arXiv  Detail & Related papers  (2020-05-01T23:20:49Z)
• A Survey on Causal Inference [64.45536158710014]
  Causal inference is a critical research topic across many domains, such as statistics, computer science, education, public policy and economics. Various causal effect estimation methods for observational data have sprung up.
  arXiv  Detail & Related papers  (2020-02-05T21:35:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.