Polyjuice: Automated, General-purpose Counterfactual Generation
- URL: http://arxiv.org/abs/2101.00288v1
- Date: Fri, 1 Jan 2021 18:34:22 GMT
- Title: Polyjuice: Automated, General-purpose Counterfactual Generation
- Authors: Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel S. Weld
- Abstract summary: We propose to disentangle counterfactual generation from its use cases, i.e., gather general-purpose counterfactuals first, and then select them for specific applications.
We frame the automated counterfactual generation as text generation, and finetune GPT-2 into a generator, Polyjuice, which produces fluent and diverse counterfactuals.
- Score: 37.152326506591876
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Counterfactual examples have been shown to be useful for many applications,
including calibrating, evaluating, and explaining model decision boundaries.
However, previous methods for generating such counterfactual examples have been
tightly tailored to a specific application, used a limited range of linguistic
patterns, or are hard to scale. We propose to disentangle counterfactual
generation from its use cases, i.e., gather general-purpose counterfactuals
first, and then select them for specific applications. We frame the automated
counterfactual generation as text generation, and finetune GPT-2 into a
generator, Polyjuice, which produces fluent and diverse counterfactuals. Our
method also allows control over where perturbations happen and what they do. We
show Polyjuice supports multiple use cases: by generating diverse
counterfactuals for humans to label, Polyjuice helps produce high-quality
datasets for model training and evaluation, requiring 40% less human effort.
When used to generate explanations, Polyjuice helps augment feature attribution
methods to reveal models' erroneous behaviors.
Related papers
- Zero-shot LLM-guided Counterfactual Generation for Text [15.254775341371364]
We propose a structured way to utilize large language models (LLMs) as general purpose counterfactual example generators.
We demonstrate the efficacy of LLMs as zero-shot counterfactual generators in evaluating and explaining black-box NLP models.
arXiv Detail & Related papers (2024-05-08T03:57:45Z) - Generative Multi-modal Models are Good Class-Incremental Learners [51.5648732517187]
We propose a novel generative multi-modal model (GMM) framework for class-incremental learning.
Our approach directly generates labels for images using an adapted generative model.
Under the Few-shot CIL setting, we have improved by at least 14% accuracy over all the current state-of-the-art methods with significantly less forgetting.
arXiv Detail & Related papers (2024-03-27T09:21:07Z) - The Integer Linear Programming Inference Cookbook [108.82092464025231]
This survey is meant to guide the reader through the process of framing a new inference problem as an instance of an integer linear program.
At the end, we will see two worked examples to illustrate the use of these recipes.
arXiv Detail & Related papers (2023-06-30T23:33:11Z) - Twist Decoding: Diverse Generators Guide Each Other [116.20780037268801]
We introduce Twist decoding, a simple and general inference algorithm that generates text while benefiting from diverse models.
Our method does not assume the vocabulary, tokenization or even generation order is shared.
arXiv Detail & Related papers (2022-05-19T01:27:53Z) - $\textit{latent}$-GLAT: Glancing at Latent Variables for Parallel Text
Generation [65.29170569821093]
parallel text generation has received widespread attention due to its success in generation efficiency.
In this paper, we propose $textitlatent$-GLAT, which employs the discrete latent variables to capture word categorical information.
Experiment results show that our method outperforms strong baselines without the help of an autoregressive model.
arXiv Detail & Related papers (2022-04-05T07:34:12Z) - Reframing Human-AI Collaboration for Generating Free-Text Explanations [46.29832336779188]
We consider the task of generating free-text explanations using a small number of human-written examples.
We find that crowdworkers often prefer explanations generated by GPT-3 to crowdsourced human-written explanations.
We create a pipeline that combines GPT-3 with a supervised filter that incorporates humans-in-the-loop via binary acceptability judgments.
arXiv Detail & Related papers (2021-12-16T07:31:37Z) - Evaluating the Disentanglement of Deep Generative Models through
Manifold Topology [66.06153115971732]
We present a method for quantifying disentanglement that only uses the generative model.
We empirically evaluate several state-of-the-art models across multiple datasets.
arXiv Detail & Related papers (2020-06-05T20:54:11Z) - Generating Natural Adversarial Hyperspectral examples with a modified
Wasserstein GAN [0.0]
We present a new method which is able to generate natural adversarial examples from the true data following the second paradigm.
We provide a proof of concept of our method by generating adversarial hyperspectral signatures on a remote sensing dataset.
arXiv Detail & Related papers (2020-01-27T07:32:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.