Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference

• URL: http://arxiv.org/abs/2106.03983v1
• Date: Mon, 7 Jun 2021 22:00:18 GMT
• Title: Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference
• Authors: Hai Hu, He Zhou, Zuoyu Tian, Yiwen Zhang, Yina Ma, Yanting Li, Yixin Nie, Kyle Richardson
• Abstract summary: We investigate the cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference (NLI) To better understand linguistic transfer, we created 4 categories of challenge and adversarial tasks for Chinese. We find that cross-lingual models trained on English NLI do transfer well across our Chinese tasks.
• Score: 11.096793445651313
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multilingual transformers (XLM, mT5) have been shown to have remarkable transfer skills in zero-shot settings. Most transfer studies, however, rely on automatically translated resources (XNLI, XQuAD), making it hard to discern the particular linguistic knowledge that is being transferred, and the role of expert annotated monolingual datasets when developing task-specific models. We investigate the cross-lingual transfer abilities of XLM-R for Chinese and English natural language inference (NLI), with a focus on the recent large-scale Chinese dataset OCNLI. To better understand linguistic transfer, we created 4 categories of challenge and adversarial tasks (totaling 17 new datasets) for Chinese that build on several well-known resources for English (e.g., HANS, NLI stress-tests). We find that cross-lingual models trained on English NLI do transfer well across our Chinese tasks (e.g., in 3/4 of our challenge categories, they perform as well/better than the best monolingual models, even on 3/5 uniquely Chinese linguistic phenomena such as idioms, pro drop). These results, however, come with important caveats: cross-lingual models often perform best when trained on a mixture of English and high-quality monolingual NLI data (OCNLI), and are often hindered by automatically translated resources (XNLI-zh). For many phenomena, all models continue to struggle, highlighting the need for our new diagnostics to help benchmark Chinese and cross-lingual models. All new datasets/code are released at https://github.com/huhailinguist/ChineseNLIProbing.

Related papers

• Dynamic data sampler for cross-language transfer learning in large language models [34.464472766868106]
  ChatFlow is a cross-language transfer-based Large Language Models (LLMs) We employ a mix of Chinese, English, and parallel corpus to continuously train the LLaMA2 model. Experimental results demonstrate that our approach accelerates model convergence and achieves superior performance.
  arXiv  Detail & Related papers  (2024-05-17T08:40:51Z)
• Zero-shot Cross-lingual Transfer without Parallel Corpus [6.937772043639308]
  We propose a novel approach to conduct zero-shot cross-lingual transfer with a pre-trained model. It consists of a Bilingual Task Fitting module that applies task-related bilingual information alignment. A self-training module generates pseudo soft and hard labels for unlabeled data and utilizes them to conduct self-training.
  arXiv  Detail & Related papers  (2023-10-07T07:54:22Z)
• Self-Augmentation Improves Zero-Shot Cross-Lingual Transfer [92.80671770992572]
  Cross-lingual transfer is a central task in multilingual NLP. Earlier efforts on this task use parallel corpora, bilingual dictionaries, or other annotated alignment data. We propose a simple yet effective method, SALT, to improve the zero-shot cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-09-19T19:30:56Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Efficiently Aligned Cross-Lingual Transfer Learning for Conversational Tasks using Prompt-Tuning [98.60739735409243]
  Cross-lingual transfer of language models trained on high-resource languages like English has been widely studied for many NLP tasks. We introduce XSGD for cross-lingual alignment pretraining, a parallel and large-scale multilingual conversation dataset. To facilitate aligned cross-lingual representations, we develop an efficient prompt-tuning-based method for learning alignment prompts.
  arXiv  Detail & Related papers  (2023-04-03T18:46:01Z)
• XNLI 2.0: Improving XNLI dataset and performance on Cross Lingual Understanding (XLU) [0.0]
  We focus on improving the original XNLI dataset by re-translating the MNLI dataset in all of the 14 different languages present in XNLI. We also perform experiments by training models in all 15 languages and analyzing their performance on the task of natural language inference.
  arXiv  Detail & Related papers  (2023-01-16T17:24:57Z)
• Languages You Know Influence Those You Learn: Impact of Language Characteristics on Multi-Lingual Text-to-Text Transfer [4.554080966463776]
  Multi-lingual language models (LM) have been remarkably successful in enabling natural language tasks in low-resource languages. We try to better understand how such models, specifically mT5, transfer *any* linguistic and semantic knowledge across languages. A key finding of this work is that similarity of syntax, morphology and phonology are good predictors of cross-lingual transfer.
  arXiv  Detail & Related papers  (2022-12-04T07:22:21Z)
• Crosslingual Generalization through Multitask Finetuning [80.8822603322471]
  Multitask prompted finetuning (MTF) has been shown to help large language models generalize to new tasks in a zero-shot setting. We apply MTF to the pretrained multilingual BLOOM and mT5 model families to produce finetuned variants called BLOOMZ and mT0. We find finetuning large multilingual language models on English tasks with English prompts allows for task generalization to non-English languages.
  arXiv  Detail & Related papers  (2022-11-03T13:19:32Z)
• OCNLI: Original Chinese Natural Language Inference [21.540733910984006]
  We present the first large-scale NLI dataset (consisting of 56,000 annotated sentence pairs) for Chinese called the Original Chinese Natural Language Inference dataset (OCNLI) Unlike recent attempts at extending NLI to other languages, our dataset does not rely on any automatic translation or non-expert annotation. We establish several baseline results on our dataset using state-of-the-art pre-trained models for Chinese, and find even the best performing models to be far outpaced by human performance.
  arXiv  Detail & Related papers  (2020-10-12T04:25:48Z)
• FILTER: An Enhanced Fusion Method for Cross-lingual Language Understanding [85.29270319872597]
  We propose an enhanced fusion method that takes cross-lingual data as input for XLM finetuning. During inference, the model makes predictions based on the text input in the target language and its translation in the source language. To tackle this issue, we propose an additional KL-divergence self-teaching loss for model training, based on auto-generated soft pseudo-labels for translated text in the target language.
  arXiv  Detail & Related papers  (2020-09-10T22:42:15Z)
• From Zero to Hero: On the Limitations of Zero-Shot Cross-Lingual Transfer with Multilingual Transformers [62.637055980148816]
  Massively multilingual transformers pretrained with language modeling objectives have become a de facto default transfer paradigm for NLP. We show that cross-lingual transfer via massively multilingual transformers is substantially less effective in resource-lean scenarios and for distant languages.
  arXiv  Detail & Related papers  (2020-05-01T22:04:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.