Plug and Play Counterfactual Text Generation for Model Robustness
- URL: http://arxiv.org/abs/2206.10429v1
- Date: Tue, 21 Jun 2022 14:25:21 GMT
- Title: Plug and Play Counterfactual Text Generation for Model Robustness
- Authors: Nishtha Madaan, Srikanta Bedathur, Diptikalyan Saha
- Abstract summary: We introduce CASPer, a plug-and-play counterfactual generation framework.
We show that CASPer effectively generates counterfactual text that follow the steering provided by an attribute model.
We also show that the generated counterfactuals can be used for augmenting the training data and thereby fixing and making the test model more robust.
- Score: 12.517365153658028
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Generating counterfactual test-cases is an important backbone for testing NLP
models and making them as robust and reliable as traditional software. In
generating the test-cases, a desired property is the ability to control the
test-case generation in a flexible manner to test for a large variety of
failure cases and to explain and repair them in a targeted manner. In this
direction, significant progress has been made in the prior works by manually
writing rules for generating controlled counterfactuals. However, this approach
requires heavy manual supervision and lacks the flexibility to easily introduce
new controls. Motivated by the impressive flexibility of the plug-and-play
approach of PPLM, we propose bringing the framework of plug-and-play to
counterfactual test case generation task. We introduce CASPer, a plug-and-play
counterfactual generation framework to generate test cases that satisfy goal
attributes on demand. Our plug-and-play model can steer the test case
generation process given any attribute model without requiring
attribute-specific training of the model. In experiments, we show that CASPer
effectively generates counterfactual text that follow the steering provided by
an attribute model while also being fluent, diverse and preserving the original
content. We also show that the generated counterfactuals from CASPer can be
used for augmenting the training data and thereby fixing and making the test
model more robust.
Related papers
- Test-Time Model Adaptation with Only Forward Passes [68.11784295706995]
Test-time adaptation has proven effective in adapting a given trained model to unseen test samples with potential distribution shifts.
We propose a test-time Forward-Optimization Adaptation (FOA) method.
FOA runs on quantized 8-bit ViT, outperforms gradient-based TENT on full-precision 32-bit ViT, and achieves an up to 24-fold memory reduction on ImageNet-C.
arXiv Detail & Related papers (2024-04-02T05:34:33Z) - Test Generation Strategies for Building Failure Models and Explaining
Spurious Failures [4.995172162560306]
Test inputs fail not only when the system under test is faulty but also when the inputs are invalid or unrealistic.
We propose to build failure models for inferring interpretable rules on test inputs that cause spurious failures.
We show that our proposed surrogate-assisted approach generates failure models with an average accuracy of 83%.
arXiv Detail & Related papers (2023-12-09T18:36:15Z) - A General Verification Framework for Dynamical and Control Models via Certificate Synthesis [54.959571890098786]
We provide a framework to encode system specifications and define corresponding certificates.
We present an automated approach to formally synthesise controllers and certificates.
Our approach contributes to the broad field of safe learning for control, exploiting the flexibility of neural networks.
arXiv Detail & Related papers (2023-09-12T09:37:26Z) - Focused Prefix Tuning for Controllable Text Generation [19.88484696133778]
We propose focused prefix tuning(FPT) to mitigate the problem and to enable the control to focus on the desired attribute.
Experimental results show that FPT can achieve better control accuracy and text fluency than baseline models in single-attribute control tasks.
arXiv Detail & Related papers (2023-06-01T06:00:43Z) - Benchmarking Model Predictive Control Algorithms in Building Optimization Testing Framework (BOPTEST) [40.17692290400862]
We present a data-driven modeling and control framework for physics-based building emulators.
Our approach consists of: (a) Offline training of differentiable surrogate models that accelerate model evaluations, provide cost-effective gradients, and maintain good predictive accuracy for the receding horizon in Model Predictive Control (MPC)
We extensively evaluate the modeling and control performance using multiple surrogate models and optimization frameworks across various test cases available in the Building Optimization Testing Framework (BOPTEST)
arXiv Detail & Related papers (2023-01-31T06:55:19Z) - Learning to Increase the Power of Conditional Randomization Tests [8.883733362171032]
The model-X conditional randomization test is a generic framework for conditional independence testing.
We introduce novel model-fitting schemes that are designed to explicitly improve the power of model-X tests.
arXiv Detail & Related papers (2022-07-03T12:29:25Z) - Controllable Text Generation with Neurally-Decomposed Oracle [91.18959622763055]
We propose a framework to control auto-regressive generation models with NeurAlly-Decomposed Oracle (NADO)
We present a closed-form optimal solution to incorporate the token-level guidance into the base model for controllable generation.
arXiv Detail & Related papers (2022-05-27T20:17:53Z) - TTAPS: Test-Time Adaption by Aligning Prototypes using Self-Supervision [70.05605071885914]
We propose a novel modification of the self-supervised training algorithm SwAV that adds the ability to adapt to single test samples.
We show the success of our method on the common benchmark dataset CIFAR10-C.
arXiv Detail & Related papers (2022-05-18T05:43:06Z) - Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
Test-time adaptation seeks to tackle potential distribution shifts between training and testing data.
We propose an active sample selection criterion to identify reliable and non-redundant samples.
We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
arXiv Detail & Related papers (2022-04-06T06:39:40Z) - MEMO: Test Time Robustness via Adaptation and Augmentation [131.28104376280197]
We study the problem of test time robustification, i.e., using the test input to improve model robustness.
Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions.
We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable.
arXiv Detail & Related papers (2021-10-18T17:55:11Z) - CAT-Gen: Improving Robustness in NLP Models via Controlled Adversarial
Text Generation [20.27052525082402]
We present a Controlled Adversarial Text Generation (CAT-Gen) model that generates adversarial texts through controllable attributes.
Experiments on real-world NLP datasets demonstrate that our method can generate more diverse and fluent adversarial texts.
arXiv Detail & Related papers (2020-10-05T21:07:45Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.