PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation

• URL: http://arxiv.org/abs/2305.16701v1
• Date: Fri, 26 May 2023 07:42:38 GMT
• Title: PIP: Parse-Instructed Prefix for Syntactically Controlled Paraphrase Generation
• Authors: Yixin Wan, Kuan-Hao Huang, Kai-Wei Chang
• Abstract summary: Parse-Instructed Prefix (PIP) is a novel adaptation of prefix-tuning to tune large pre-trained language models. In contrast to traditional fine-tuning methods for this task, PIP is a compute-efficient alternative with 10 times less learnable parameters.
• Score: 61.05254852400895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Syntactically controlled paraphrase generation requires language models to generate paraphrases for sentences according to specific syntactic structures. Existing fine-tuning methods for this task are costly as all the parameters of the model need to be updated during the training process. Inspired by recent studies on parameter-efficient learning, we propose Parse-Instructed Prefix (PIP), a novel adaptation of prefix-tuning to tune large pre-trained language models on syntactically controlled paraphrase generation task in a low-data setting with significantly less training cost. We introduce two methods to instruct a model's encoder prefix to capture syntax-related knowledge: direct initiation (PIP-Direct) and indirect optimization (PIP-Indirect). In contrast to traditional fine-tuning methods for this task, PIP is a compute-efficient alternative with 10 times less learnable parameters. Compared to existing prefix-tuning methods, PIP excels at capturing syntax control information, achieving significantly higher performance at the same level of learnable parameter count.

Related papers

• Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods [69.36397993451742]
  This work introduces Context-aware Prompt Tuning (CPT), a method inspired by ICL, PT, and adversarial attacks. We modify specific context tokens, considering the unique structure of input and output formats. Inspired by adversarial attacks, we adjust the input based on the labels present in the context, focusing on minimizing, rather than maximizing, the loss.
  arXiv  Detail & Related papers  (2024-10-22T17:45:47Z)
• Parameter-Efficient Fine-Tuning of Large Language Models using Semantic Knowledge Tuning [0.08795040582681389]
  Large Language Models (LLMs) are gaining significant popularity in recent years for specialized tasks using prompts. We propose a novel method called Semantic Knowledge Tuning (SK-Tuning) for prompt and prefix tuning that employs meaningful words instead of random tokens. Our experimental results show that SK-Tuning exhibits faster training times, fewer parameters, and superior performance on tasks such as text classification and understanding.
  arXiv  Detail & Related papers  (2024-10-11T07:55:09Z)
• RIFF: Learning to Rephrase Inputs for Few-shot Fine-tuning of Language Models [4.085425430499285]
  We explore the impact of altering the input text of the original task in conjunction with parameter-efficient fine-tuning methods. To most effectively rewrite the input text, we train a few-shot paraphrase model with a Maximum-Marginal Likelihood objective. We show that enriching data with paraphrases at train and test time enhances the performance beyond what can be achieved with parameter-efficient fine-tuning alone.
  arXiv  Detail & Related papers  (2024-03-04T17:58:09Z)
• Approximated Prompt Tuning for Vision-Language Pre-trained Models [54.326232586461614]
  In vision-language pre-trained models, prompt tuning often requires a large number of learnable tokens to bridge the gap between the pre-training and downstream tasks. We propose a novel Approximated Prompt Tuning (APT) approach towards efficient VL transfer learning.
  arXiv  Detail & Related papers  (2023-06-27T05:43:47Z)
• Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptation [11.960178399478718]
  Large pretrained language models (PLMs) are often domain- or task-adapted via fine-tuning or prompting. Instead, we prepare PLMs for data- and parameter-efficient adaptation by learning to learn the difference between general and adapted PLMs.
  arXiv  Detail & Related papers  (2022-07-07T18:00:22Z)
• Instance-wise Prompt Tuning for Pretrained Language Models [72.74916121511662]
  Instance-wise Prompt Tuning (IPT) is the first prompt learning paradigm that injects knowledge from the input data instances to the prompts. IPT significantly outperforms task-based prompt learning methods, and achieves comparable performance to conventional finetuning with only 0.5% - 1.5% of tuned parameters.
  arXiv  Detail & Related papers  (2022-06-04T10:08:50Z)
• IDPG: An Instance-Dependent Prompt Generation Method [58.45110542003139]
  Prompt tuning is a new, efficient NLP transfer learning paradigm that adds a task-specific prompt in each input instance during the model training stage. We propose a conditional prompt generation method to generate prompts for each input instance.
  arXiv  Detail & Related papers  (2022-04-09T15:45:27Z)
• Prefix-Tuning: Optimizing Continuous Prompts for Generation [85.6357778621526]
  Fine-tuning is the de facto way to leverage large pretrained language models to perform downstream tasks. We propose prefix-tuning, a lightweight alternative to fine-tuning for natural language generation tasks. We find that by learning only 0.1% of the parameters, prefix-tuning obtains comparable performance in the full data setting.
  arXiv  Detail & Related papers  (2021-01-01T08:00:36Z)
• WARP: Word-level Adversarial ReProgramming [13.08689221166729]
  In many applications it is preferable to tune much smaller sets of parameters, so that the majority of parameters can be shared across multiple tasks. We present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. We show that this approach outperforms other methods with a similar number of trainable parameters on SST-2 and MNLI datasets.
  arXiv  Detail & Related papers  (2021-01-01T00:41:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.