Learning to Transfer Prompts for Text Generation

• URL: http://arxiv.org/abs/2205.01543v1
• Date: Tue, 3 May 2022 14:53:48 GMT
• Title: Learning to Transfer Prompts for Text Generation
• Authors: Junyi Li, Tianyi Tang, Jian-Yun Nie, Ji-Rong Wen and Wayne Xin Zhao
• Abstract summary: We propose a novel prompt-based method (PTG) for text generation in a transferable setting. First, PTG learns a set of source prompts for various source generation tasks and then transfers these prompts as target prompts to perform target generation tasks. In extensive experiments, PTG yields competitive or better results than fine-tuning methods.
• Score: 97.64625999380425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pretrained language models (PLMs) have made remarkable progress in text generation tasks via fine-tuning. While, it is challenging to fine-tune PLMs in a data-scarce situation. Therefore, it is non-trivial to develop a general and lightweight model that can adapt to various text generation tasks based on PLMs. To fulfill this purpose, the recent prompt-based learning offers a potential solution. In this paper, we improve this technique and propose a novel prompt-based method (PTG) for text generation in a transferable setting. First, PTG learns a set of source prompts for various source generation tasks and then transfers these prompts as target prompts to perform target generation tasks. To consider both task- and instance-level information, we design an adaptive attention mechanism to derive the target prompts. For each data instance, PTG learns a specific target prompt by attending to highly relevant source prompts. In extensive experiments, PTG yields competitive or better results than fine-tuning methods. We release our source prompts as an open resource, where users can add or reuse them to improve new text generation tasks for future research. Code and data can be available at https://github.com/RUCAIBox/Transfer-Prompts-for-Text-Generation.

Related papers

• Learning to Prompt with Text Only Supervision for Vision-Language Models [107.282881515667]
  One branch of methods adapts CLIP by learning prompts using visual information. An alternative approach resorts to training-free methods by generating class descriptions from large language models. We propose to combine the strengths of both streams by learning prompts using only text data.
  arXiv  Detail & Related papers  (2024-01-04T18:59:49Z)
• Text-driven Prompt Generation for Vision-Language Models in Federated Learning [24.005620820818756]
  Our work proposes Federated Text-driven Prompt Generation (FedTPG) FedTPG learns a unified prompt generation network across multiple remote clients in a scalable manner. Our comprehensive empirical evaluations on nine diverse image classification datasets show that our method is superior to existing federated prompt learning methods.
  arXiv  Detail & Related papers  (2023-10-09T19:57:24Z)
• Deliberate then Generate: Enhanced Prompting Framework for Text Generation [70.10319005141888]
  Deliberate then Generate (DTG) prompting framework consists of error detection instructions and candidates that may contain errors. We conduct extensive experiments on 20+ datasets across 7 text generation tasks, including summarization, translation, dialogue, and more. We show that DTG consistently outperforms existing prompting methods and achieves state-of-the-art performance on multiple text generation tasks.
  arXiv  Detail & Related papers  (2023-05-31T13:23:04Z)
• TegTok: Augmenting Text Generation via Task-specific and Open-world Knowledge [83.55215993730326]
  We propose augmenting TExt Generation via Task-specific and Open-world Knowledge (TegTok) in a unified framework. Our model selects knowledge entries from two types of knowledge sources through dense retrieval and then injects them into the input encoding and output decoding stages respectively.
  arXiv  Detail & Related papers  (2022-03-16T10:37:59Z)
• Context-Tuning: Learning Contextualized Prompts for Natural Language Generation [52.835877179365525]
  We propose a novel continuous prompting approach, called Context-Tuning, to fine-tuning PLMs for natural language generation. Firstly, the prompts are derived based on the input text, so that they can elicit useful knowledge from PLMs for generation. Secondly, to further enhance the relevance of the generated text to the inputs, we utilize continuous inverse prompting to refine the process of natural language generation.
  arXiv  Detail & Related papers  (2022-01-21T12:35:28Z)
• A Survey of Pretrained Language Models Based Text Generation [97.64625999380425]
  Text Generation aims to produce plausible and readable text in human language from input data. Deep learning has greatly advanced this field by neural generation models, especially the paradigm of pretrained language models (PLMs) Grounding text generation on PLMs is seen as a promising direction in both academia and industry.
  arXiv  Detail & Related papers  (2022-01-14T01:44:58Z)
• Control Prefixes for Text Generation [17.682443394199375]
  We propose a dynamic method, Control Prefixes, which allows for the inclusion of conditional input-dependent information in each prompt. We present state-of-the-art results on several data-to-text datasets, including WebNLG.
  arXiv  Detail & Related papers  (2021-10-15T19:32:17Z)
• Zero-shot Learning by Generating Task-specific Adapters [38.452434222367515]
  We introduce Hypter, a framework that improves zero-shot transferability by training a hypernetwork to generate task-specific adapters from task descriptions. This formulation enables learning at task level, and greatly reduces the number of parameters by using light-weight adapters.
  arXiv  Detail & Related papers  (2021-01-02T10:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.