Establishing Interlingua in Multilingual Language Models

• URL: http://arxiv.org/abs/2109.01207v1
• Date: Thu, 2 Sep 2021 20:53:14 GMT
• Title: Establishing Interlingua in Multilingual Language Models
• Authors: Maksym Del, Mark Fishel
• Abstract summary: We show that different languages do converge to a shared space in large multilingual language models. We extend our analysis to 28 diverse languages and find that the interlingual space exhibits a particular structure similar to the linguistic relatedness of languages.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large multilingual language models show remarkable zero-shot cross-lingual transfer performance on a range of tasks. Follow-up works hypothesized that these models internally project representations of different languages into a shared interlingual space. However, they produced contradictory results. In this paper, we correct %one of the previous works the famous prior work claiming that "BERT is not an Interlingua" and show that with the proper choice of sentence representation different languages actually do converge to a shared space in such language models. Furthermore, we demonstrate that this convergence pattern is robust across four measures of correlation similarity and six mBERT-like models. We then extend our analysis to 28 diverse languages and find that the interlingual space exhibits a particular structure similar to the linguistic relatedness of languages. We also highlight a few outlier languages that seem to fail to converge to the shared space. The code for replicating our results is available at the following URL: https://github.com/maksym-del/interlingua.

Related papers

• The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments [57.273662221547056]
  In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. We observe that the existence of a predominant language during training boosts the performance of less frequent languages. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
  arXiv  Detail & Related papers  (2024-04-11T17:58:05Z)
• Counterfactually Probing Language Identity in Multilingual Models [15.260518230218414]
  We use AlterRep, a method of counterfactual probing, to explore the internal structure of multilingual models. We find that, given a template in Language X, pushing towards Language Y systematically increases the probability of Language Y words.
  arXiv  Detail & Related papers  (2023-10-29T01:21:36Z)
• The Less the Merrier? Investigating Language Representation in Multilingual Models [8.632506864465501]
  We investigate the linguistic representation of different languages in multilingual models. We observe from our experiments that community-centered models perform better at distinguishing between languages in the same family for low-resource languages.
  arXiv  Detail & Related papers  (2023-10-20T02:26:34Z)
• Exploring Anisotropy and Outliers in Multilingual Language Models for Cross-Lingual Semantic Sentence Similarity [64.18762301574954]
  Previous work has shown that the representations output by contextual language models are more anisotropic than static type embeddings. This seems to be true for both monolingual and multilingual models, although much less work has been done on the multilingual context. We investigate outlier dimensions and their relationship to anisotropy in multiple pre-trained multilingual language models.
  arXiv  Detail & Related papers  (2023-06-01T09:01:48Z)
• Discovering Language-neutral Sub-networks in Multilingual Language Models [15.94622051535847]
  Language neutrality of multilingual models is a function of the overlap between language-encoding sub-networks of these models. Using mBERT as a foundation, we employ the lottery ticket hypothesis to discover sub-networks that are individually optimized for various languages and tasks. We conclude that mBERT is comprised of a language-neutral sub-network shared among many languages, along with multiple ancillary language-specific sub-networks.
  arXiv  Detail & Related papers  (2022-05-25T11:35:41Z)
• A Massively Multilingual Analysis of Cross-linguality in Shared Embedding Space [61.18554842370824]
  In cross-lingual language models, representations for many different languages live in the same space. We compute a task-based measure of cross-lingual alignment in the form of bitext retrieval performance. We examine a range of linguistic, quasi-linguistic, and training-related features as potential predictors of these alignment metrics.
  arXiv  Detail & Related papers  (2021-09-13T21:05:37Z)
• Examining Cross-lingual Contextual Embeddings with Orthogonal Structural Probes [0.2538209532048867]
  A novel Orthogonal Structural Probe (Limisiewicz and Marevcek, 2021) allows us to answer this question for specific linguistic features. We evaluate syntactic (UD) and lexical (WordNet) structural information encoded inmBERT's contextual representations for nine diverse languages. We successfully apply our findings to zero-shot and few-shot cross-lingual parsing.
  arXiv  Detail & Related papers  (2021-09-10T15:03:11Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models [96.32118305166412]
  We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We find that languages which are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts.
  arXiv  Detail & Related papers  (2020-12-31T14:11:00Z)
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora [63.5286019659504]
  We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus. Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence.
  arXiv  Detail & Related papers  (2020-10-27T22:24:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.