Improving Classifier Robustness through Active Generation of Pairwise Counterfactuals

• URL: http://arxiv.org/abs/2305.13535v1
• Date: Mon, 22 May 2023 23:19:01 GMT
• Title: Improving Classifier Robustness through Active Generation of Pairwise Counterfactuals
• Authors: Ananth Balashankar, Xuezhi Wang, Yao Qin, Ben Packer, Nithum Thain, Jilin Chen, Ed H. Chi, Alex Beutel
• Abstract summary: We present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals. We show that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels.
• Score: 22.916599410472102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Counterfactual Data Augmentation (CDA) is a commonly used technique for improving robustness in natural language classifiers. However, one fundamental challenge is how to discover meaningful counterfactuals and efficiently label them, with minimal human labeling cost. Most existing methods either completely rely on human-annotated labels, an expensive process which limits the scale of counterfactual data, or implicitly assume label invariance, which may mislead the model with incorrect labels. In this paper, we present a novel framework that utilizes counterfactual generative models to generate a large number of diverse counterfactuals by actively sampling from regions of uncertainty, and then automatically label them with a learned pairwise classifier. Our key insight is that we can more correctly label the generated counterfactuals by training a pairwise classifier that interpolates the relationship between the original example and the counterfactual. We demonstrate that with a small amount of human-annotated counterfactual data (10%), we can generate a counterfactual augmentation dataset with learned labels, that provides an 18-20% improvement in robustness and a 14-21% reduction in errors on 6 out-of-domain datasets, comparable to that of a fully human-annotated counterfactual dataset for both sentiment classification and question paraphrase tasks.

Related papers

• Coupled Confusion Correction: Learning from Crowds with Sparse Annotations [43.94012824749425]
  Confusion matrices learned by two models can be corrected by the distilled data from the other. We cluster the annotator groups'' who share similar expertise so that their confusion matrices could be corrected together.
  arXiv  Detail & Related papers  (2023-12-12T14:47:26Z)
• Increasing Diversity While Maintaining Accuracy: Text Data Generation with Large Language Models and Human Interventions [30.464763055981933]
  Large language models (LLMs) can be used to generate text data for training and evaluating other models. We explore human-AI partnerships to facilitate high diversity and accuracy in LLM-based text data generation.
  arXiv  Detail & Related papers  (2023-06-07T04:27:09Z)
• Beyond Active Learning: Leveraging the Full Potential of Human Interaction via Auto-Labeling, Human Correction, and Human Verification [3.58439716487063]
  Active Learning (AL) is a human-in-the-loop framework to interactively and adaptively label data instances. We propose CLARIFIER, an Interactive Learning framework that admits more effective use of human interaction by leveraging the reduced cost of verification.
  arXiv  Detail & Related papers  (2023-06-02T05:40:11Z)
• Dynamic Conceptional Contrastive Learning for Generalized Category Discovery [76.82327473338734]
  Generalized category discovery (GCD) aims to automatically cluster partially labeled data. Unlabeled data contain instances that are not only from known categories of the labeled data but also from novel categories. One effective way for GCD is applying self-supervised learning to learn discriminate representation for unlabeled data. We propose a Dynamic Conceptional Contrastive Learning framework, which can effectively improve clustering accuracy.
  arXiv  Detail & Related papers  (2023-03-30T14:04:39Z)
• Trustable Co-label Learning from Multiple Noisy Annotators [68.59187658490804]
  Supervised deep learning depends on massive accurately annotated examples. A typical alternative is learning from multiple noisy annotators. This paper proposes a data-efficient approach, called emphTrustable Co-label Learning (TCL)
  arXiv  Detail & Related papers  (2022-03-08T16:57:00Z)
• Debiased Pseudo Labeling in Self-Training [77.83549261035277]
  Deep neural networks achieve remarkable performances on a wide range of tasks with the aid of large-scale labeled datasets. To mitigate the requirement for labeled data, self-training is widely used in both academia and industry by pseudo labeling on readily-available unlabeled data. We propose Debiased, in which the generation and utilization of pseudo labels are decoupled by two independent heads.
  arXiv  Detail & Related papers  (2022-02-15T02:14:33Z)
• Robustness to Spurious Correlations in Text Classification via Automatically Generated Counterfactuals [8.827892752465958]
  We propose to train a robust text classifier by augmenting the training data with automatically generated counterfactual data. We show that the robust classifier makes meaningful and trustworthy predictions by emphasizing causal features and de-emphasizing non-causal features.
  arXiv  Detail & Related papers  (2020-12-18T03:57:32Z)
• How to trust unlabeled data? Instance Credibility Inference for Few-Shot Learning [47.21354101796544]
  This paper presents a statistical approach, dubbed Instance Credibility Inference (ICI) to exploit the support of unlabeled instances for few-shot visual recognition. We rank the credibility of pseudo-labeled instances along the regularization path of their corresponding incidental parameters, and the most trustworthy pseudo-labeled examples are preserved as the augmented labeled instances.
  arXiv  Detail & Related papers  (2020-07-15T03:38:09Z)
• Improving Face Recognition by Clustering Unlabeled Faces in the Wild [77.48677160252198]
  We propose a novel identity separation method based on extreme value theory. It greatly reduces the problems caused by overlapping-identity label noise. Experiments on both controlled and real settings demonstrate our method's consistent improvements.
  arXiv  Detail & Related papers  (2020-07-14T12:26:50Z)
• Classify and Generate Reciprocally: Simultaneous Positive-Unlabelled Learning and Conditional Generation with Extra Data [77.31213472792088]
  The scarcity of class-labeled data is a ubiquitous bottleneck in many machine learning problems. We address this problem by leveraging Positive-Unlabeled(PU) classification and the conditional generation with extra unlabeled data. We present a novel training framework to jointly target both PU classification and conditional generation when exposed to extra data.
  arXiv  Detail & Related papers  (2020-06-14T08:27:40Z)
• Adversarial Self-Supervised Contrastive Learning [62.17538130778111]
  Existing adversarial learning approaches mostly use class labels to generate adversarial samples that lead to incorrect predictions. We propose a novel adversarial attack for unlabeled data, which makes the model confuse the instance-level identities of the perturbed data samples. We present a self-supervised contrastive learning framework to adversarially train a robust neural network without labeled data.
  arXiv  Detail & Related papers  (2020-06-13T08:24:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.