On Robust Prefix-Tuning for Text Classification
- URL: http://arxiv.org/abs/2203.10378v1
- Date: Sat, 19 Mar 2022 18:52:47 GMT
- Title: On Robust Prefix-Tuning for Text Classification
- Authors: Zonghan Yang, Yang Liu
- Abstract summary: We propose a robust prefix-tuning framework that preserves the efficiency and modularity of prefix-tuning.
Our framework substantially improves robustness over several strong baselines against five textual attacks of different types.
- Score: 16.08753509741376
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recently, prefix-tuning has gained increasing attention as a
parameter-efficient finetuning method for large-scale pretrained language
models. The method keeps the pretrained models fixed and only updates the
prefix token parameters for each downstream task. Despite being lightweight and
modular, prefix-tuning still lacks robustness to textual adversarial attacks.
However, most currently developed defense techniques necessitate auxiliary
model update and storage, which inevitably hamper the modularity and low
storage of prefix-tuning. In this work, we propose a robust prefix-tuning
framework that preserves the efficiency and modularity of prefix-tuning. The
core idea of our framework is leveraging the layerwise activations of the
language model by correctly-classified training data as the standard for
additional prefix finetuning. During the test phase, an extra batch-level
prefix is tuned for each batch and added to the original prefix for robustness
enhancement. Extensive experiments on three text classification benchmarks show
that our framework substantially improves robustness over several strong
baselines against five textual attacks of different types while maintaining
comparable accuracy on clean texts. We also interpret our robust prefix-tuning
framework from the optimal control perspective and pose several directions for
future research.
Related papers
- Towards Infinite-Long Prefix in Transformer [18.24137806007111]
We study the ability of Prompting and context-based fine-tuning methods to match the performance of full parameter fine-tuning.
We implement an algorithm that only needs to introduce and fine-tune a few extra trainable parameters instead of an infinite-long prefix.
Our method achieves superior or competitive performance compared to existing methods like full parameters fine-tuning, P-Tuning V2, and LoRA.
arXiv Detail & Related papers (2024-06-20T06:56:35Z) - PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase
Generation [61.05254852400895]
Parse-Instructed Prefix (PIP) is a novel adaptation of prefix-tuning to tune large pre-trained language models.
In contrast to traditional fine-tuning methods for this task, PIP is a compute-efficient alternative with 10 times less learnable parameters.
arXiv Detail & Related papers (2023-05-26T07:42:38Z) - Towards Adaptive Prefix Tuning for Parameter-Efficient Language Model
Fine-tuning [32.84435258519842]
We propose Adaptive Prefix Tuning (APT) to adjust the prefix in terms of both fine-grained token level and coarse-grained layer level with a gate mechanism.
Experiments on the SuperGLUE and NER datasets show the effectiveness of APT.
arXiv Detail & Related papers (2023-05-24T14:51:01Z) - CCPrefix: Counterfactual Contrastive Prefix-Tuning for Many-Class
Classification [57.62886091828512]
We propose a brand-new prefix-tuning method, Counterfactual Contrastive Prefix-tuning (CCPrefix) for many-class classification.
Basically, an instance-dependent soft prefix, derived from fact-counterfactual pairs in the label space, is leveraged to complement the language verbalizers in many-class classification.
arXiv Detail & Related papers (2022-11-11T03:45:59Z) - Inducer-tuning: Connecting Prefix-tuning and Adapter-tuning [53.72897232951918]
We show that inducer-tuning can close the performance gap between prefix-tuning and fine-tuning.
We suggest a new variant of prefix-tuning -- textitinducer-tuning, which shares the exact mechanism as prefix-tuning while leveraging the residual form found in adapter-tuning.
arXiv Detail & Related papers (2022-10-26T04:39:42Z) - Adversarial Self-Attention for Language Understanding [89.265747130584]
This paper proposes textitAdversarial Self-Attention mechanism (ASA).
ASA adversarially reconstructs the Transformer attentions and facilitates model training from contaminated model structures.
For fine-tuning, ASA-empowered models consistently outweigh naive models by a large margin considering both generalization and robustness.
arXiv Detail & Related papers (2022-06-25T09:18:10Z) - Discourse-Aware Prompt Design for Text Generation [13.835916386769474]
We show that prompt based conditional text generation can be improved with simple and efficient methods.
First, we show that a higher-level discourse structure of human written text can be modelled with textithierarchical blocking on prefix parameters.
Second, we propose sparse prefix tuning by introducing textitattention sparsity on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function.
arXiv Detail & Related papers (2021-12-10T18:15:44Z) - Prefix-Tuning: Optimizing Continuous Prompts for Generation [85.6357778621526]
Fine-tuning is the de facto way to leverage large pretrained language models to perform downstream tasks.
We propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks.
We find that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting.
arXiv Detail & Related papers (2021-01-01T08:00:36Z) - Selecting Informative Contexts Improves Language Model Finetuning [66.26521454263343]
We present a general fine-tuning method that we call information gain filtration.
During fine-tuning, a secondary learner selects informative examples and skips uninformative ones.
We show that our method has consistent improvement across datasets, fine-tuning tasks, and language model architectures.
arXiv Detail & Related papers (2020-05-01T02:01:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.