Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction

• URL: http://arxiv.org/abs/2105.06965v1
• Date: Fri, 14 May 2021 17:11:55 GMT
• Title: Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Prediction
• Authors: Shauli Ravfogel, Grusha Prasad, Tal Linzen, Yoav Goldberg
• Abstract summary: We show that BERT uses information about RC spans during agreement prediction using the linguistically strategy. We also found that counterfactual representations generated for a specific RC subtype influenced the number prediction in sentences with other RC subtypes, suggesting that information about RC boundaries was encoded abstractly in BERT's representation.
• Score: 61.4913233397155
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When language models process syntactically complex sentences, do they use abstract syntactic information present in these sentences in a manner that is consistent with the grammar of English, or do they rely solely on a set of heuristics? We propose a method to tackle this question, AlterRep. For any linguistic feature in the sentence, AlterRep allows us to generate counterfactual representations by altering how this feature is encoded, while leaving all other aspects of the original representation intact. Then, by measuring the change in a models' word prediction with these counterfactual representations in different sentences, we can draw causal conclusions about the contexts in which the model uses the linguistic feature (if any). Applying this method to study how BERT uses relative clause (RC) span information, we found that BERT uses information about RC spans during agreement prediction using the linguistically strategy. We also found that counterfactual representations generated for a specific RC subtype influenced the number prediction in sentences with other RC subtypes, suggesting that information about RC boundaries was encoded abstractly in BERT's representation.

Related papers

• Intervention Lens: from Representation Surgery to String Counterfactuals [106.98481791980367]
  Interventions targeting the representation space of language models (LMs) have emerged as an effective means to influence model behavior. We give a method to convert representation counterfactuals into string counterfactuals. The resulting counterfactuals can be used to mitigate bias in classification through data augmentation.
  arXiv  Detail & Related papers  (2024-02-17T18:12:02Z)
• Panoptic Scene Graph Generation with Semantics-Prototype Learning [23.759498629378772]
  Panoptic Scene Graph Generation (PSG) parses objects and predicts their relationships (predicate) to connect human language and visual scenes. Different language preferences of annotators and semantic overlaps between predicates lead to biased predicate annotations. We propose a novel framework named ADTrans to adaptively transfer biased predicate annotations to informative and unified ones.
  arXiv  Detail & Related papers  (2023-07-28T14:04:06Z)
• Conjunct Resolution in the Face of Verbal Omissions [51.220650412095665]
  We propose a conjunct resolution task that operates directly on the text and makes use of a split-and-rephrase paradigm in order to recover the missing elements in the coordination structure. We curate a large dataset, containing over 10K examples of naturally-occurring verbal omissions with crowd-sourced annotations. We train various neural baselines for this task, and show that while our best method obtains decent performance, it leaves ample space for improvement.
  arXiv  Detail & Related papers  (2023-05-26T08:44:02Z)
• Enriching Relation Extraction with OpenIE [70.52564277675056]
  Relation extraction (RE) is a sub-discipline of information extraction (IE) In this work, we explore how recent approaches for open information extraction (OpenIE) may help to improve the task of RE. Our experiments over two annotated corpora, KnowledgeNet and FewRel, demonstrate the improved accuracy of our enriched models.
  arXiv  Detail & Related papers  (2022-12-19T11:26:23Z)
• Understanding Translationese in Cross-Lingual Summarization [106.69566000567598]
  Cross-lingual summarization (MS) aims at generating a concise summary in a different target language. To collect large-scale CLS data, existing datasets typically involve translation in their creation. In this paper, we first confirm that different approaches of constructing CLS datasets will lead to different degrees of translationese.
  arXiv  Detail & Related papers  (2022-12-14T13:41:49Z)
• Probing for targeted syntactic knowledge through grammatical error detection [13.653209309144593]
  We propose grammatical error detection as a diagnostic probe to evaluate pre-trained English language models. We leverage public annotated training data from both English second language learners and Wikipedia edits. We find that masked language models linearly encode information relevant to the detection of SVA errors, while the autoregressive models perform on par with our baseline.
  arXiv  Detail & Related papers  (2022-10-28T16:01:25Z)
• Multilingual Extraction and Categorization of Lexical Collocations with Graph-aware Transformers [86.64972552583941]
  We put forward a sequence tagging BERT-based model enhanced with a graph-aware transformer architecture, which we evaluate on the task of collocation recognition in context. Our results suggest that explicitly encoding syntactic dependencies in the model architecture is helpful, and provide insights on differences in collocation typification in English, Spanish and French.
  arXiv  Detail & Related papers  (2022-05-23T16:47:37Z)
• Exploring the Role of BERT Token Representations to Explain Sentence Probing Results [15.652077779677091]
  We show that BERT tends to encode meaningful knowledge in specific token representations. This allows the model to detect syntactic and semantic abnormalities and to distinctively separate grammatical number and tense subspaces.
  arXiv  Detail & Related papers  (2021-04-03T20:40:42Z)
• Picking BERT's Brain: Probing for Linguistic Dependencies in Contextualized Embeddings Using Representational Similarity Analysis [13.016284599828232]
  We investigate the degree to which a verb embedding encodes the verb's subject, a pronoun embedding encodes the pronoun's antecedent, and a full-sentence representation encodes the sentence's head word. In all cases, we show that BERT's contextualized embeddings reflect the linguistic dependency being studied, and that BERT encodes these dependencies to a greater degree than it encodes less linguistically-salient controls.
  arXiv  Detail & Related papers  (2020-11-24T13:19:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.