Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations

• URL: http://arxiv.org/abs/2404.02452v1
• Date: Wed, 3 Apr 2024 04:40:57 GMT
• Title: Adaptive Cross-lingual Text Classification through In-Context One-Shot Demonstrations
• Authors: Emilio Villa-Cueva, A. Pastor López-Monroy, Fernando Sánchez-Vega, Thamar Solorio,
• Abstract summary: We exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer in the classification task by introducing In-Context Cross-lingual Transfer (IC-XLT) The novel concept involves training a model to learn from context examples and subsequently adapting it during inference to a target language by prepending a One-Shot context demonstration in that language. Our results show that IC-XLT successfully leverages target-language examples to improve the cross-lingual capabilities of the evaluated mT5 model, outperforming prompt-based models in the Zero and Few-shot scenarios adapted through fine-tuning.
• Score: 47.89819316477715
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Zero-Shot Cross-lingual Transfer (ZS-XLT) utilizes a model trained in a source language to make predictions in another language, often with a performance loss. To alleviate this, additional improvements can be achieved through subsequent adaptation using examples in the target language. In this paper, we exploit In-Context Tuning (ICT) for One-Shot Cross-lingual transfer in the classification task by introducing In-Context Cross-lingual Transfer (IC-XLT). The novel concept involves training a model to learn from context examples and subsequently adapting it during inference to a target language by prepending a One-Shot context demonstration in that language. Our results show that IC-XLT successfully leverages target-language examples to improve the cross-lingual capabilities of the evaluated mT5 model, outperforming prompt-based models in the Zero and Few-shot scenarios adapted through fine-tuning. Moreover, we show that when source-language data is limited, the fine-tuning framework employed for IC-XLT performs comparably to prompt-based fine-tuning with significantly more training data in the source language.

Related papers

• X-SNS: Cross-Lingual Transfer Prediction through Sub-Network Similarity [19.15213046428148]
  Cross-lingual transfer (XLT) is an ability of multilingual language models that preserves their performance on a task to a significant extent when evaluated in languages that were not included in the fine-tuning process. We propose the utilization of sub-network similarity between two languages as a proxy for predicting the compatibility of the languages in the context of XLT.
  arXiv  Detail & Related papers  (2023-10-26T05:39:49Z)
• Improving the Cross-Lingual Generalisation in Visual Question Answering [40.86774711775718]
  multilingual vision-language pretrained models show poor cross-lingual generalisation when applied to non-English data. In this work, we explore the poor performance of these models on a zero-shot cross-lingual visual question answering (VQA) task. We improve cross-lingual transfer with three strategies: (1) we introduce a linguistic prior objective to augment the cross-entropy loss with a similarity-based loss to guide the model during training, (2) we learn a task-specific subnetwork that improves cross-lingual generalisation and reduces variance without model modification, and (3) we augment training examples using synthetic code
  arXiv  Detail & Related papers  (2022-09-07T08:07:43Z)
• Multi-Level Contrastive Learning for Cross-Lingual Alignment [35.33431650608965]
  Cross-language pre-trained models such as multilingual BERT (mBERT) have achieved significant performance in various cross-lingual downstream NLP tasks. This paper proposes a multi-level contrastive learning framework to further improve the cross-lingual ability of pre-trained models.
  arXiv  Detail & Related papers  (2022-02-26T07:14:20Z)
• CLIN-X: pre-trained language models and a study on cross-task transfer for concept extraction in the clinical domain [22.846469609263416]
  We introduce the pre-trained CLIN-X (Clinical XLM-R) language models and show how CLIN-X outperforms other pre-trained transformer models. Our studies reveal stable model performance despite a lack of annotated data with improvements of up to 47 F1 points when only 250 labeled sentences are available. Our results highlight the importance of specialized language models as CLIN-X for concept extraction in non-standard domains.
  arXiv  Detail & Related papers  (2021-12-16T10:07:39Z)
• Factorized Neural Transducer for Efficient Language Model Adaptation [51.81097243306204]
  We propose a novel model, factorized neural Transducer, by factorizing the blank and vocabulary prediction. It is expected that this factorization can transfer the improvement of the standalone language model to the Transducer for speech recognition. We demonstrate that the proposed factorized neural Transducer yields 15% to 20% WER improvements when out-of-domain text data is used for language model adaptation.
  arXiv  Detail & Related papers  (2021-09-27T15:04:00Z)
• Cross-lingual Transferring of Pre-trained Contextualized Language Models [73.97131976850424]
  We propose a novel cross-lingual model transferring framework for PrLMs: TreLM. To handle the symbol order and sequence length differences between languages, we propose an intermediate TRILayer" structure. We show the proposed framework significantly outperforms language models trained from scratch with limited data in both performance and efficiency.
  arXiv  Detail & Related papers  (2021-07-27T06:51:13Z)
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model [58.27176041092891]
  Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements. We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
  arXiv  Detail & Related papers  (2020-11-23T16:00:42Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment [71.53159402053392]
  We propose a regularization approach to align word-level and sentence-level representations across languages without any external resource. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios.
  arXiv  Detail & Related papers  (2020-09-30T08:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.