To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding?

• URL: http://arxiv.org/abs/2011.05007v1
• Date: Tue, 10 Nov 2020 09:59:24 GMT
• Title: To What Degree Can Language Borders Be Blurred In BERT-based Multilingual Spoken Language Understanding?
• Authors: Quynh Do, Judith Gaspers, Tobias Roding, Melanie Bradford
• Abstract summary: We show that although a BERT-based multilingual Spoken Language Understanding (SLU) model works substantially well even on distant language groups, there is still a gap to the ideal multilingual performance. We propose a novel BERT-based adversarial model architecture to learn language-shared and language-specific representations for multilingual SLU.
• Score: 7.245261469258502
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper addresses the question as to what degree a BERT-based multilingual Spoken Language Understanding (SLU) model can transfer knowledge across languages. Through experiments we will show that, although it works substantially well even on distant language groups, there is still a gap to the ideal multilingual performance. In addition, we propose a novel BERT-based adversarial model architecture to learn language-shared and language-specific representations for multilingual SLU. Our experimental results prove that the proposed model is capable of narrowing the gap to the ideal multilingual performance.

Related papers

• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Multilingual BERT has an accent: Evaluating English influences on fluency in multilingual models [23.62852626011989]
  We show that grammatical structures in higher-resource languages bleed into lower-resource languages. We show this bias via a novel method for comparing the fluency of multilingual models to the fluency of monolingual Spanish and Greek models.
  arXiv  Detail & Related papers  (2022-10-11T17:06:38Z)
• Generalizing Multimodal Pre-training into Multilingual via Language Acquisition [54.69707237195554]
  English-based Vision-Language Pre-training has achieved great success in various downstream tasks. Some efforts have been taken to generalize this success to non-English languages through Multilingual Vision-Language Pre-training. We propose a textbfMultitextbfLingual textbfAcquisition (MLA) framework that can easily generalize a monolingual Vision-Language Pre-training model into multilingual.
  arXiv  Detail & Related papers  (2022-05-29T08:53:22Z)
• Exploring Teacher-Student Learning Approach for Multi-lingual Speech-to-Intent Classification [73.5497360800395]
  We develop an end-to-end system that supports multiple languages. We exploit knowledge from a pre-trained multi-lingual natural language processing model.
  arXiv  Detail & Related papers  (2021-09-28T04:43:11Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• Towards Fully Bilingual Deep Language Modeling [1.3455090151301572]
  We consider whether it is possible to pre-train a bilingual model for two remotely related languages without compromising performance at either language. We create a Finnish-English bilingual BERT model and evaluate its performance on datasets used to evaluate the corresponding monolingual models. Our bilingual model performs on par with Google's original English BERT on GLUE and nearly matches the performance of monolingual Finnish BERT on a range of Finnish NLP tasks.
  arXiv  Detail & Related papers  (2020-10-22T12:22:50Z)
• Looking for Clues of Language in Multilingual BERT to Improve Cross-lingual Generalization [56.87201892585477]
  Token embeddings in multilingual BERT (m-BERT) contain both language and semantic information. We control the output languages of multilingual BERT by manipulating the token embeddings.
  arXiv  Detail & Related papers  (2020-10-20T05:41:35Z)
• Identifying Necessary Elements for BERT's Multilinguality [4.822598110892846]
  multilingual BERT (mBERT) yields high quality multilingual representations and enables effective zero-shot transfer. We aim to identify architectural properties of BERT and linguistic properties of languages that are necessary for BERT to become multilingual.
  arXiv  Detail & Related papers  (2020-05-01T14:27:14Z)
• A Study of Cross-Lingual Ability and Language-specific Information in Multilingual BERT [60.9051207862378]
  multilingual BERT works remarkably well on cross-lingual transfer tasks. Datasize and context window size are crucial factors to the transferability. There is a computationally cheap but effective approach to improve the cross-lingual ability of multilingual BERT.
  arXiv  Detail & Related papers  (2020-04-20T11:13:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.