Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt

• URL: http://arxiv.org/abs/2202.11451v1
• Date: Wed, 23 Feb 2022 11:57:52 GMT
• Title: Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt
• Authors: Lianzhe Huang, Shuming Ma, Dongdong Zhang, Furu Wei and Houfeng Wang
• Abstract summary: We propose a novel model that uses a unified prompt for all languages, called UniPrompt. The unified prompt is computation by a multilingual PLM to produce language-independent representation. Our proposed methods can significantly outperform the strong baselines across different languages.
• Score: 98.26682501616024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt-based tuning has been proven effective for pretrained language models (PLMs). While most of the existing work focuses on the monolingual prompts, we study the multilingual prompts for multilingual PLMs, especially in the zero-shot cross-lingual setting. To alleviate the effort of designing different prompts for multiple languages, we propose a novel model that uses a unified prompt for all languages, called UniPrompt. Different from the discrete prompts and soft prompts, the unified prompt is model-based and language-agnostic. Specifically, the unified prompt is initialized by a multilingual PLM to produce language-independent representation, after which is fused with the text input. During inference, the prompts can be pre-computed so that no extra computation cost is needed. To collocate with the unified prompt, we propose a new initialization method for the target label word to further improve the model's transferability across languages. Extensive experiments show that our proposed methods can significantly outperform the strong baselines across different languages. We will release data and code to facilitate future research.

Related papers

• Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator [104.63314132355221]
  Cross-lingual transfer with prompt learning has shown promising effectiveness. We propose a novel framework, Multilingual Prompt Translator (MPT) MPT is more prominent compared with vanilla prompting when transferring to languages quite distinct from source language.
  arXiv  Detail & Related papers  (2024-03-19T03:35:18Z)
• Few-Shot Cross-Lingual Transfer for Prompting Large Language Models in Low-Resource Languages [0.0]
  "prompting" is where a user provides a description of a task and some completed examples of the task to a PLM as context before prompting the PLM to perform the task on a new example. We consider three methods: few-shot prompting (prompt), language-adaptive fine-tuning (LAFT), and neural machine translation (translate) We find that translate and prompt settings are a compute-efficient and cost-effective method of few-shot prompting for the selected low-resource languages.
  arXiv  Detail & Related papers  (2024-03-09T21:36:13Z)
• Helping Language Models Learn More: Multi-dimensional Task Prompt for Few-shot Tuning [36.14688633670085]
  We propose MTPrompt, a multi-dimensional task prompt learning method based on task-related object, summary, and task description information. By automatically building and searching for appropriate prompts, our proposed MTPrompt achieves the best results on few-shot samples setting and five different datasets.
  arXiv  Detail & Related papers  (2023-12-13T10:00:44Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer [52.46740830977898]
  Cross-lingual natural language inference is a fundamental problem in cross-lingual language understanding. We propose a novel Soft prompt learning framework with the Multilingual Verbalizer (SoftMV) for XNLI.
  arXiv  Detail & Related papers  (2023-05-22T06:31:29Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.