Polyjuice: Automated, General-purpose Counterfactual Generation

• URL: http://arxiv.org/abs/2101.00288v1
• Date: Fri, 1 Jan 2021 18:34:22 GMT
• Title: Polyjuice: Automated, General-purpose Counterfactual Generation
• Authors: Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, Daniel S. Weld
• Abstract summary: We propose to disentangle counterfactual generation from its use cases, i.e., gather general-purpose counterfactuals first, and then select them for specific applications. We frame the automated counterfactual generation as text generation, and finetune GPT-2 into a generator, Polyjuice, which produces fluent and diverse counterfactuals.
• Score: 37.152326506591876
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Counterfactual examples have been shown to be useful for many applications, including calibrating, evaluating, and explaining model decision boundaries. However, previous methods for generating such counterfactual examples have been tightly tailored to a specific application, used a limited range of linguistic patterns, or are hard to scale. We propose to disentangle counterfactual generation from its use cases, i.e., gather general-purpose counterfactuals first, and then select them for specific applications. We frame the automated counterfactual generation as text generation, and finetune GPT-2 into a generator, Polyjuice, which produces fluent and diverse counterfactuals. Our method also allows control over where perturbations happen and what they do. We show Polyjuice supports multiple use cases: by generating diverse counterfactuals for humans to label, Polyjuice helps produce high-quality datasets for model training and evaluation, requiring 40% less human effort. When used to generate explanations, Polyjuice helps augment feature attribution methods to reveal models' erroneous behaviors.

Related papers

• Improving Conditional Level Generation using Automated Validation in Match-3 Games [39.887603099741696]
  This paper proposes Avalon, a novel method to improve models that learn from existing level designs. We use a conditional variational autoencoder to generate layouts for match-3 levels. We quantitatively evaluate our approach by comparing it to an ablated model without difficulty conditioning.
  arXiv  Detail & Related papers  (2024-09-10T09:07:47Z)
• Zero-shot LLM-guided Counterfactual Generation for Text [15.254775341371364]
  We propose a structured way to utilize large language models (LLMs) as general purpose counterfactual example generators. We demonstrate the efficacy of LLMs as zero-shot counterfactual generators in evaluating and explaining black-box NLP models.
  arXiv  Detail & Related papers  (2024-05-08T03:57:45Z)
• Manipulating and Mitigating Generative Model Biases without Retraining [49.60774626839712]
  We propose a dynamic and computationally efficient manipulation of T2I model biases by exploiting their rich language embedding spaces without model retraining. We show that leveraging foundational vector algebra allows for a convenient control over language model embeddings to shift T2I model outputs. As a by-product, this control serves as a form of precise prompt engineering to generate images which are generally implausible using regular text prompts.
  arXiv  Detail & Related papers  (2024-04-03T07:33:30Z)
• Generative Multi-modal Models are Good Class-Incremental Learners [51.5648732517187]
  We propose a novel generative multi-modal model (GMM) framework for class-incremental learning. Our approach directly generates labels for images using an adapted generative model. Under the Few-shot CIL setting, we have improved by at least 14% accuracy over all the current state-of-the-art methods with significantly less forgetting.
  arXiv  Detail & Related papers  (2024-03-27T09:21:07Z)
• The Integer Linear Programming Inference Cookbook [108.82092464025231]
  This survey is meant to guide the reader through the process of framing a new inference problem as an instance of an integer linear program. At the end, we will see two worked examples to illustrate the use of these recipes.
  arXiv  Detail & Related papers  (2023-06-30T23:33:11Z)
• Twist Decoding: Diverse Generators Guide Each Other [116.20780037268801]
  We introduce Twist decoding, a simple and general inference algorithm that generates text while benefiting from diverse models. Our method does not assume the vocabulary, tokenization or even generation order is shared.
  arXiv  Detail & Related papers  (2022-05-19T01:27:53Z)
• Reframing Human-AI Collaboration for Generating Free-Text Explanations [46.29832336779188]
  We consider the task of generating free-text explanations using a small number of human-written examples. We find that crowdworkers often prefer explanations generated by GPT-3 to crowdsourced human-written explanations. We create a pipeline that combines GPT-3 with a supervised filter that incorporates humans-in-the-loop via binary acceptability judgments.
  arXiv  Detail & Related papers  (2021-12-16T07:31:37Z)
• Evaluating the Disentanglement of Deep Generative Models through Manifold Topology [66.06153115971732]
  We present a method for quantifying disentanglement that only uses the generative model. We empirically evaluate several state-of-the-art models across multiple datasets.
  arXiv  Detail & Related papers  (2020-06-05T20:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.