Romanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibition

• URL: http://arxiv.org/abs/2304.11350v2
• Date: Mon, 8 May 2023 16:54:03 GMT
• Title: Romanian Multiword Expression Detection Using Multilingual Adversarial Training and Lateral Inhibition
• Authors: Andrei-Marius Avram, Verginica Barbu Mititelu and Dumitru-Clementin Cercel
• Abstract summary: This paper describes our improvements in automatically identifying Romanian multiword expressions on the corpus released for the PARSEME v1.2 shared task. Our approach assumes a multilingual perspective based on the recently introduced lateral inhibition layer and adversarial training to boost the performance of the employed multilingual language models.
• Score: 0.17188280334580194
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Multiword expressions are a key ingredient for developing large-scale and linguistically sound natural language processing technology. This paper describes our improvements in automatically identifying Romanian multiword expressions on the corpus released for the PARSEME v1.2 shared task. Our approach assumes a multilingual perspective based on the recently introduced lateral inhibition layer and adversarial training to boost the performance of the employed multilingual language models. With the help of these two methods, we improve the F1-score of XLM-RoBERTa by approximately 2.7% on unseen multiword expressions, the main task of the PARSEME 1.2 edition. In addition, our results can be considered SOTA performance, as they outperform the previous results on Romanian obtained by the participants in this competition.

Related papers

• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [90.73517523001149]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct. We propose different training strategies to build powerful xMR LLMs, named MathOctopus, notably outperform conventional open-source LLMs.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• The Effectiveness of Intermediate-Task Training for Code-Switched Natural Language Understanding [15.54831836850549]
  We propose the use of bilingual intermediate pretraining as a reliable technique to derive performance gains on three different NLP tasks using code-switched text. We achieve substantial absolute improvements of 7.87%, 20.15%, and 10.99%, on the mean accuracies and F1 scores over previous state-of-the-art systems. We show consistent performance gains on four different code-switched language-pairs (Hindi-English, Spanish-English, Tamil-English and Malayalam-English) for SA.
  arXiv  Detail & Related papers  (2021-07-21T08:10:59Z)
• Contrastive Learning for Many-to-many Multilingual Neural Machine Translation [16.59039088482523]
  Existing multilingual machine translation approaches mainly focus on English-centric directions. We aim to build a many-to-many translation system with an emphasis on the quality of non-English language directions.
  arXiv  Detail & Related papers  (2021-05-20T03:59:45Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)
• Towards Multi-Sense Cross-Lingual Alignment of Contextual Embeddings [41.148892848434585]
  We propose a novel framework to align contextual embeddings at the sense level by leveraging cross-lingual signal from bilingual dictionaries only. We operationalize our framework by first proposing a novel sense-aware cross entropy loss to model word senses explicitly. We then propose a sense alignment objective on top of the sense-aware cross entropy loss for cross-lingual model pretraining, and pretrain cross-lingual models for several language pairs.
  arXiv  Detail & Related papers  (2021-03-11T04:55:35Z)
• Mixed-Lingual Pre-training for Cross-lingual Summarization [54.4823498438831]
  Cross-lingual Summarization aims at producing a summary in the target language for an article in the source language. We propose a solution based on mixed-lingual pre-training that leverages both cross-lingual tasks like translation and monolingual tasks like masked language models. Our model achieves an improvement of 2.82 (English to Chinese) and 1.15 (Chinese to English) ROUGE-1 scores over state-of-the-art results.
  arXiv  Detail & Related papers  (2020-10-18T00:21:53Z)
• UPB at SemEval-2020 Task 9: Identifying Sentiment in Code-Mixed Social Media Texts using Transformers and Multi-Task Learning [1.7196613099537055]
  We describe the systems developed by our team for SemEval-2020 Task 9. We aim to cover two well-known code-mixed languages: Hindi-English and Spanish-English. Our approach achieves promising performance on the Hindi-English task, with an average F1-score of 0.6850. For the Spanish-English task, we obtained an average F1-score of 0.7064 ranking our team 17th out of 29 participants.
  arXiv  Detail & Related papers  (2020-09-06T17:19:18Z)
• On Learning Universal Representations Across Languages [37.555675157198145]
  We extend existing approaches to learn sentence-level representations and show the effectiveness on cross-lingual understanding and generation. Specifically, we propose a Hierarchical Contrastive Learning (HiCTL) method to learn universal representations for parallel sentences distributed in one or multiple languages. We conduct evaluations on two challenging cross-lingual tasks, XTREME and machine translation.
  arXiv  Detail & Related papers  (2020-07-31T10:58:39Z)
• CoSDA-ML: Multi-Lingual Code-Switching Data Augmentation for Zero-Shot Cross-Lingual NLP [68.2650714613869]
  We propose a data augmentation framework to generate multi-lingual code-switching data to fine-tune mBERT. Compared with the existing work, our method does not rely on bilingual sentences for training, and requires only one training process for multiple target languages.
  arXiv  Detail & Related papers  (2020-06-11T13:15:59Z)
• Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translation [81.7786241489002]
  Massively multilingual models for neural machine translation (NMT) are theoretically attractive, but often underperform bilingual models and deliver poor zero-shot translations. We argue that multilingual NMT requires stronger modeling capacity to support language pairs with varying typological characteristics. We propose random online backtranslation to enforce the translation of unseen training language pairs.
  arXiv  Detail & Related papers  (2020-04-24T17:21:32Z)
• Robust Cross-lingual Embeddings from Parallel Sentences [65.85468628136927]
  We propose a bilingual extension of the CBOW method which leverages sentence-aligned corpora to obtain robust cross-lingual word representations. Our approach significantly improves crosslingual sentence retrieval performance over all other approaches. It also achieves parity with a deep RNN method on a zero-shot cross-lingual document classification task.
  arXiv  Detail & Related papers  (2019-12-28T16:18:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.