Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models

• URL: http://arxiv.org/abs/2310.01691v2
• Date: Fri, 12 Jul 2024 17:26:08 GMT
• Title: Zero-Shot Continuous Prompt Transfer: Generalizing Task Semantics Across Language Models
• Authors: Zijun Wu, Yongkang Wu, Lili Mou,
• Abstract summary: We propose a zero-shot continuous prompt transfer method, where source prompts are encoded into relative space and the corresponding target prompts are searched for transferring to target models. Experimental results confirm the effectiveness of our method, showing that 'task semantics' in continuous prompts can be generalized across various language models.
• Score: 24.114485240965383
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Prompt tuning in natural language processing (NLP) has become an increasingly popular method for adapting large language models to specific tasks. However, the transferability of these prompts, especially continuous prompts, between different models remains a challenge. In this work, we propose a zero-shot continuous prompt transfer method, where source prompts are encoded into relative space and the corresponding target prompts are searched for transferring to target models. Experimental results confirm the effectiveness of our method, showing that 'task semantics' in continuous prompts can be generalized across various language models. Moreover, we find that combining 'task semantics' from multiple source models can further enhance the generalizability of transfer.

Related papers

• Eliciting Textual Descriptions from Representations of Continuous Prompts [11.489611613744724]
  We propose a new approach to interpret continuous prompts that elicits textual descriptions from their representations during model inference. We show our method often yields accurate task descriptions which become more faithful as task performance increases. InSPEcT can be leveraged to debug unwanted properties in continuous prompts and inform developers on ways to mitigate them.
  arXiv  Detail & Related papers  (2024-10-15T14:46:11Z)
• Exploring the Transferability of Visual Prompting for Multimodal Large Language Models [47.162575147632396]
  Transferable Visual Prompting (TVP) is a simple and effective approach to generate visual prompts that can transfer to different models and improve their performance on downstream tasks after trained on only one model. We introduce two strategies to address the issue of cross-model feature corruption of existing visual prompting methods and enhance the transferability of the learned prompts.
  arXiv  Detail & Related papers  (2024-04-17T09:39:07Z)
• Scalable Language Model with Generalized Continual Learning [58.700439919096155]
  The Joint Adaptive Re-ization (JARe) is integrated with Dynamic Task-related Knowledge Retrieval (DTKR) to enable adaptive adjustment of language models based on specific downstream tasks. Our method demonstrates state-of-the-art performance on diverse backbones and benchmarks, achieving effective continual learning in both full-set and few-shot scenarios with minimal forgetting.
  arXiv  Detail & Related papers  (2024-04-11T04:22:15Z)
• Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator [104.63314132355221]
  Cross-lingual transfer with prompt learning has shown promising effectiveness. We propose a novel framework, Multilingual Prompt Translator (MPT) MPT is more prominent compared with vanilla prompting when transferring to languages quite distinct from source language.
  arXiv  Detail & Related papers  (2024-03-19T03:35:18Z)
• UniverSLU: Universal Spoken Language Understanding for Diverse Tasks with Natural Language Instructions [64.50935101415776]
  We build a single model that jointly performs various spoken language understanding (SLU) tasks. We demonstrate the efficacy of our single multi-task learning model "UniverSLU" for 12 speech classification and sequence generation task types spanning 17 datasets and 9 languages.
  arXiv  Detail & Related papers  (2023-10-04T17:10:23Z)
• On Conditional and Compositional Language Model Differentiable Prompting [75.76546041094436]
  Prompts have been shown to be an effective method to adapt a frozen Pretrained Language Model (PLM) to perform well on downstream tasks. We propose a new model, Prompt Production System (PRopS), which learns to transform task instructions or input metadata, into continuous prompts.
  arXiv  Detail & Related papers  (2023-07-04T02:47:42Z)
• DiTTO: A Feature Representation Imitation Approach for Improving Cross-Lingual Transfer [15.062937537799005]
  languages as domains for improving zero-shot transfer. We show that our approach, DiTTO, significantly outperforms the standard zero-shot fine-tuning method. Our model enables better cross-lingual transfer than standard fine-tuning methods, even in the few-shot setting.
  arXiv  Detail & Related papers  (2023-03-04T08:42:50Z)
• Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt [98.26682501616024]
  We propose a novel model that uses a unified prompt for all languages, called UniPrompt. The unified prompt is computation by a multilingual PLM to produce language-independent representation. Our proposed methods can significantly outperform the strong baselines across different languages.
  arXiv  Detail & Related papers  (2022-02-23T11:57:52Z)
• SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer [7.2462572989580405]
  We propose a novel prompt-based transfer learning approach called SPoT: Soft Prompt Transfer. We show SPoT significantly boosts the performance of PromptTuning across many tasks. We also conduct a large-scale study on task transferability with 26 NLP tasks and 160 combinations of source-target tasks.
  arXiv  Detail & Related papers  (2021-10-15T07:35:58Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.