Robustification of Multilingual Language Models to Real-world Noise with Robust Contrastive Pretraining

• URL: http://arxiv.org/abs/2210.04782v1
• Date: Mon, 10 Oct 2022 15:40:43 GMT
• Title: Robustification of Multilingual Language Models to Real-world Noise with Robust Contrastive Pretraining
• Authors: Asa Cooper Stickland, Sailik Sengupta, Jason Krone, Saab Mansour, He He
• Abstract summary: We assess the robustness of neural models on noisy data and suggest improvements are limited to the English language. To benchmark the performance of pretrained multilingual models, we construct noisy datasets covering five languages and four NLP tasks. We propose Robust Contrastive Pretraining (RCP) to boost the zero-shot cross-lingual robustness of multilingual pretrained models.
• Score: 14.087882550564169
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Advances in neural modeling have achieved state-of-the-art (SOTA) results on public natural language processing (NLP) benchmarks, at times surpassing human performance. However, there is a gap between public benchmarks and real-world applications where noise such as typos or grammatical mistakes is abundant, resulting in degraded performance. Unfortunately, works that assess the robustness of neural models on noisy data and suggest improvements are limited to the English language. Upon analyzing noise in different languages, we observe that noise types vary across languages and thus require their own investigation. Thus, to benchmark the performance of pretrained multilingual models, we construct noisy datasets covering five languages and four NLP tasks. We see a gap in performance between clean and noisy data. After investigating ways to boost the zero-shot cross-lingual robustness of multilingual pretrained models, we propose Robust Contrastive Pretraining (RCP). RCP combines data augmentation with a contrastive loss term at the pretraining stage and achieves large improvements on noisy (& original test data) across two sentence-level classification (+3.2%) and two sequence-labeling (+10 F1-score) multilingual tasks.

Related papers

• A Comparison of Language Modeling and Translation as Multilingual Pretraining Objectives [13.581385765600265]
  Pretrained language models (PLMs) display impressive performances and have captured the attention of the NLP community. This paper proposes a comparison of multilingual pretraining objectives in a controlled methodological environment.
  arXiv  Detail & Related papers  (2024-07-22T09:16:30Z)
• READIN: A Chinese Multi-Task Benchmark with Realistic and Diverse Input Noises [87.70001456418504]
  We construct READIN: a Chinese multi-task benchmark with REalistic And Diverse Input Noises. READIN contains four diverse tasks and requests annotators to re-enter the original test data with two commonly used Chinese input methods: Pinyin input and speech input. We experiment with a series of strong pretrained language models as well as robust training methods, we find that these models often suffer significant performance drops on READIN.
  arXiv  Detail & Related papers  (2023-02-14T20:14:39Z)
• Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning [25.230786853723203]
  We propose a noise-robust cross-lingual cross-modal retrieval method for low-resource languages. We use Machine Translation to construct pseudo-parallel sentence pairs for low-resource languages. We introduce a multi-view self-distillation method to learn noise-robust target-language representations.
  arXiv  Detail & Related papers  (2022-08-26T09:32:24Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• From Good to Best: Two-Stage Training for Cross-lingual Machine Reading Comprehension [51.953428342923885]
  We develop a two-stage approach to enhance the model performance. The first stage targets at recall: we design a hard-learning (HL) algorithm to maximize the likelihood that the top-k predictions contain the accurate answer. The second stage focuses on precision: an answer-aware contrastive learning mechanism is developed to learn the fine difference between the accurate answer and other candidates.
  arXiv  Detail & Related papers  (2021-12-09T07:31:15Z)
• Understanding Model Robustness to User-generated Noisy Texts [2.958690090551675]
  In NLP, model performance often deteriorates with naturally occurring noise, such as spelling errors. We propose to model the errors statistically from grammatical-error-correction corpora.
  arXiv  Detail & Related papers  (2021-10-14T14:54:52Z)
• Learning from Multiple Noisy Augmented Data Sets for Better Cross-Lingual Spoken Language Understanding [69.40915115518523]
  Lack of training data presents a grand challenge to scaling out spoken language understanding (SLU) to low-resource languages. Various data augmentation approaches have been proposed to synthesize training data in low-resource target languages. In this paper we focus on mitigating noise in augmented data.
  arXiv  Detail & Related papers  (2021-09-03T15:44:15Z)
• Evaluating the Robustness of Neural Language Models to Input Perturbations [7.064032374579076]
  In this study, we design and implement various types of character-level and word-level perturbation methods to simulate noisy input texts. We investigate the ability of high-performance language models such as BERT, XLNet, RoBERTa, and ELMo in handling different types of input perturbations. The results suggest that language models are sensitive to input perturbations and their performance can decrease even when small changes are introduced.
  arXiv  Detail & Related papers  (2021-08-27T12:31:17Z)
• AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languages [75.08199398141744]
  We present AmericasNLI, an extension of XNLI (Conneau et al.), to 10 indigenous languages of the Americas. We conduct experiments with XLM-R, testing multiple zero-shot and translation-based approaches. We find that XLM-R's zero-shot performance is poor for all 10 languages, with an average performance of 38.62%.
  arXiv  Detail & Related papers  (2021-04-18T05:32:28Z)
• Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning [74.25168207651376]
  Fine-tuning pre-trained language models to downstream cross-lingual tasks has shown promising results. We leverage continual learning to preserve the cross-lingual ability of the pre-trained model when we fine-tune it to downstream tasks. Our methods achieve better performance than other fine-tuning baselines on the zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.
  arXiv  Detail & Related papers  (2020-04-29T14:07:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.