How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?

• URL: http://arxiv.org/abs/2109.07022v1
• Date: Tue, 14 Sep 2021 23:46:39 GMT
• Title: How Does Counterfactually Augmented Data Impact Models for Social Computing Constructs?
• Authors: Indira Sen, Mattia Samory, Fabian Floeck, Claudia Wagner, Isabelle Augenstein
• Abstract summary: We investigate the benefits of counterfactually augmented data (CAD) for social NLP models by focusing on three social computing constructs -- sentiment, sexism, and hate speech. We find that while models trained on CAD show lower in-domain performance, they generalize better out-of-domain.
• Score: 35.29235215101502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As NLP models are increasingly deployed in socially situated settings such as online abusive content detection, it is crucial to ensure that these models are robust. One way of improving model robustness is to generate counterfactually augmented data (CAD) for training models that can better learn to distinguish between core features and data artifacts. While models trained on this type of data have shown promising out-of-domain generalizability, it is still unclear what the sources of such improvements are. We investigate the benefits of CAD for social NLP models by focusing on three social computing constructs -- sentiment, sexism, and hate speech. Assessing the performance of models trained with and without CAD across different types of datasets, we find that while models trained on CAD show lower in-domain performance, they generalize better out-of-domain. We unpack this apparent discrepancy using machine explanations and find that CAD reduces model reliance on spurious features. Leveraging a novel typology of CAD to analyze their relationship with model performance, we find that CAD which acts on the construct directly or a diverse set of CAD leads to higher performance.

Related papers

• PairCFR: Enhancing Model Training on Paired Counterfactually Augmented Data through Contrastive Learning [49.60634126342945]
  Counterfactually Augmented Data (CAD) involves creating new data samples by applying minimal yet sufficient modifications to flip the label of existing data samples to other classes. Recent research reveals that training with CAD may lead models to overly focus on modified features while ignoring other important contextual information. We employ contrastive learning to promote global feature alignment in addition to learning counterfactual clues.
  arXiv  Detail & Related papers  (2024-06-09T07:29:55Z)
• ContrastCAD: Contrastive Learning-based Representation Learning for Computer-Aided Design Models [0.7373617024876725]
  We propose a contrastive learning-based approach to learning CAD models, named ContrastCAD. ContrastCAD effectively captures semantic information within the construction sequences of the CAD model. We also propose a new CAD data augmentation method, called a Random Replace and Extrude (RRE) method, to enhance the learning performance of the model.
  arXiv  Detail & Related papers  (2024-04-02T05:30:39Z)
• Fine-Tuning Enhances Existing Mechanisms: A Case Study on Entity Tracking [53.66999416757543]
  We study how fine-tuning affects the internal mechanisms implemented in language models. Fine-tuning enhances, rather than alters, the mechanistic operation of the model.
  arXiv  Detail & Related papers  (2024-02-22T18:59:24Z)
• People Make Better Edits: Measuring the Efficacy of LLM-Generated Counterfactually Augmented Data for Harmful Language Detection [35.89913036572029]
  It is imperative that NLP models are robust to spurious features. Past work has attempted to tackle such spurious features using training data augmentation. We assess if this task can be automated using generative NLP models.
  arXiv  Detail & Related papers  (2023-11-02T14:31:25Z)
• Robustness and Generalization Performance of Deep Learning Models on Cyber-Physical Systems: A Comparative Study [71.84852429039881]
  Investigation focuses on the models' ability to handle a range of perturbations, such as sensor faults and noise. We test the generalization and transfer learning capabilities of these models by exposing them to out-of-distribution (OOD) samples.
  arXiv  Detail & Related papers  (2023-06-13T12:43:59Z)
• Dataless Knowledge Fusion by Merging Weights of Language Models [51.8162883997512]
  Fine-tuning pre-trained language models has become the prevalent paradigm for building downstream NLP models. This creates a barrier to fusing knowledge across individual models to yield a better single model. We propose a dataless knowledge fusion method that merges models in their parameter space.
  arXiv  Detail & Related papers  (2022-12-19T20:46:43Z)
• AutoCAD: Automatically Generating Counterfactuals for Mitigating Shortcut Learning [70.70393006697383]
  We present AutoCAD, a fully automatic and task-agnostic CAD generation framework. In this paper, we present AutoCAD, a fully automatic and task-agnostic CAD generation framework.
  arXiv  Detail & Related papers  (2022-11-29T13:39:53Z)
• Counterfactually Augmented Data and Unintended Bias: The Case of Sexism and Hate Speech Detection [35.29235215101502]
  Over-relying on core features may lead to unintended model bias. We test models for sexism and hate speech detection on challenging data. Using a diverse set of CAD -- construct-driven and construct-agnostic -- reduces such unintended bias.
  arXiv  Detail & Related papers  (2022-05-09T12:39:26Z)
• An Investigation of the (In)effectiveness of Counterfactually Augmented Data [10.316235366821111]
  We show that while counterfactually-augmented data (CAD) is effective at identifying robust features, it may prevent the model from learning unperturbed robust features. Our results show that the lack of perturbation diversity in current CAD datasets limits its effectiveness on OOD generalization.
  arXiv  Detail & Related papers  (2021-07-01T21:46:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.