Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

• URL: http://arxiv.org/abs/2006.00995v3
• Date: Fri, 19 Feb 2021 09:01:59 GMT
• Title: Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals
• Authors: Yanai Elazar, Shauli Ravfogel, Alon Jacovi, Yoav Goldberg
• Abstract summary: We point out the inability to infer behavioral conclusions from probing results. We offer an alternative method that focuses on how the information is being used, rather than on what information is encoded.
• Score: 53.484562601127195
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A growing body of work makes use of probing to investigate the working of neural models, often considered black boxes. Recently, an ongoing debate emerged surrounding the limitations of the probing paradigm. In this work, we point out the inability to infer behavioral conclusions from probing results and offer an alternative method that focuses on how the information is being used, rather than on what information is encoded. Our method, Amnesic Probing, follows the intuition that the utility of a property for a given task can be assessed by measuring the influence of a causal intervention that removes it from the representation. Equipped with this new analysis tool, we can ask questions that were not possible before, e.g. is part-of-speech information important for word prediction? We perform a series of analyses on BERT to answer these types of questions. Our findings demonstrate that conventional probing performance is not correlated to task importance, and we call for increased scrutiny of claims that draw behavioral or causal conclusions from probing results.

Related papers

• Unsupervised Pairwise Causal Discovery on Heterogeneous Data using Mutual Information Measures [49.1574468325115]
  Causal Discovery is a technique that tackles the challenge by analyzing the statistical properties of the constituent variables. We question the current (possibly misleading) baseline results on the basis that they were obtained through supervised learning. In consequence, we approach this problem in an unsupervised way, using robust Mutual Information measures.
  arXiv  Detail & Related papers  (2024-08-01T09:11:08Z)
• CausalGym: Benchmarking causal interpretability methods on linguistic tasks [52.61917615039112]
  We use CausalGym to benchmark the ability of interpretability methods to causally affect model behaviour. We study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods. We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena.
  arXiv  Detail & Related papers  (2024-02-19T21:35:56Z)
• Explaining Veracity Predictions with Evidence Summarization: A Multi-Task Model Approach [1.223779595809275]
  We propose a multi-task explainable neural model for misinformation detection. Specifically, this work formulates an explanation generation process of the model's veracity prediction as a text summarization problem. The performance of the proposed model is discussed on publicly available datasets and the findings are evaluated with related studies.
  arXiv  Detail & Related papers  (2024-02-09T14:39:20Z)
• Clarify When Necessary: Resolving Ambiguity Through Interaction with LMs [58.620269228776294]
  We propose a task-agnostic framework for resolving ambiguity by asking users clarifying questions. We evaluate systems across three NLP applications: question answering, machine translation and natural language inference. We find that intent-sim is robust, demonstrating improvements across a wide range of NLP tasks and LMs.
  arXiv  Detail & Related papers  (2023-11-16T00:18:50Z)
• Disentangled Representation for Causal Mediation Analysis [25.114619307838602]
  Causal mediation analysis is a method that is often used to reveal direct and indirect effects. Deep learning shows promise in mediation analysis, but the current methods only assume latent confounders that affect treatment, mediator and outcome simultaneously. We propose the Disentangled Mediation Analysis Variational AutoEncoder (DMAVAE), which disentangles the representations of latent confounders into three types to accurately estimate the natural direct effect, natural indirect effect and total effect.
  arXiv  Detail & Related papers  (2023-02-19T23:37:17Z)
• Active Bayesian Causal Inference [72.70593653185078]
  We propose Active Bayesian Causal Inference (ABCI), a fully-Bayesian active learning framework for integrated causal discovery and reasoning. ABCI jointly infers a posterior over causal models and queries of interest. We show that our approach is more data-efficient than several baselines that only focus on learning the full causal graph.
  arXiv  Detail & Related papers  (2022-06-04T22:38:57Z)
• Naturalistic Causal Probing for Morpho-Syntax [76.83735391276547]
  We suggest a naturalistic strategy for input-level intervention on real world data in Spanish. Using our approach, we isolate morpho-syntactic features from counfounders in sentences. We apply this methodology to analyze causal effects of gender and number on contextualized representations extracted from pre-trained models.
  arXiv  Detail & Related papers  (2022-05-14T11:47:58Z)
• An introduction to causal reasoning in health analytics [2.199093822766999]
  We will try to highlight some of the drawbacks that may arise in traditional machine learning and statistical approaches to analyze the observational data. We will demonstrate the applications of causal inference in tackling some common machine learning issues.
  arXiv  Detail & Related papers  (2021-05-10T20:25:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.