Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer

• URL: http://arxiv.org/abs/2305.12761v1
• Date: Mon, 22 May 2023 06:31:29 GMT
• Title: Enhancing Cross-lingual Natural Language Inference by Soft Prompting with Multilingual Verbalizer
• Authors: Shuang Li, Xuming Hu, Aiwei Liu, Yawen Yang, Fukun Ma, Philip S. Yu, Lijie Wen
• Abstract summary: Cross-lingual natural language inference is a fundamental problem in cross-lingual language understanding. We propose a novel Soft prompt learning framework with the Multilingual Verbalizer (SoftMV) for XNLI.
• Score: 52.46740830977898
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-lingual natural language inference is a fundamental problem in cross-lingual language understanding. Many recent works have used prompt learning to address the lack of annotated parallel corpora in XNLI. However, these methods adopt discrete prompting by simply translating the templates to the target language and need external expert knowledge to design the templates. Besides, discrete prompts of human-designed template words are not trainable vectors and can not be migrated to target languages in the inference stage flexibly. In this paper, we propose a novel Soft prompt learning framework with the Multilingual Verbalizer (SoftMV) for XNLI. SoftMV first constructs cloze-style question with soft prompts for the input sample. Then we leverage bilingual dictionaries to generate an augmented multilingual question for the original question. SoftMV adopts a multilingual verbalizer to align the representations of original and augmented multilingual questions into the same semantic space with consistency regularization. Experimental results on XNLI demonstrate that SoftMV can achieve state-of-the-art performance and significantly outperform the previous methods under the few-shot and full-shot cross-lingual transfer settings.

Related papers

• Cross-Lingual Transfer for Natural Language Inference via Multilingual Prompt Translator [104.63314132355221]
  Cross-lingual transfer with prompt learning has shown promising effectiveness. We propose a novel framework, Multilingual Prompt Translator (MPT) MPT is more prominent compared with vanilla prompting when transferring to languages quite distinct from source language.
  arXiv  Detail & Related papers  (2024-03-19T03:35:18Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Zero-shot Cross-lingual Transfer of Prompt-based Tuning with a Unified Multilingual Prompt [98.26682501616024]
  We propose a novel model that uses a unified prompt for all languages, called UniPrompt. The unified prompt is computation by a multilingual PLM to produce language-independent representation. Our proposed methods can significantly outperform the strong baselines across different languages.
  arXiv  Detail & Related papers  (2022-02-23T11:57:52Z)
• Syntax-augmented Multilingual BERT for Cross-lingual Transfer [37.99210035238424]
  This work shows that explicitly providing language syntax and training mBERT helps cross-lingual transfer. Experiment results show that syntax-augmented mBERT improves cross-lingual transfer on popular benchmarks.
  arXiv  Detail & Related papers  (2021-06-03T21:12:50Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.