LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization

• URL: http://arxiv.org/abs/2401.06034v6
• Date: Fri, 04 Oct 2024 06:32:07 GMT
• Title: LinguAlchemy: Fusing Typological and Geographical Elements for Unseen Language Generalization
• Authors: Muhammad Farid Adilazuarda, Samuel Cahyawijaya, Alham Fikri Aji, Genta Indra Winata, Ayu Purwarianti,
• Abstract summary: LinguAlchemy is a regularization method that incorporates various linguistic information covering typological, geographical, and phylogenetic features. Our LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages.
• Score: 35.12566667582262
• License:
• Abstract: Pretrained language models (PLMs) have become remarkably adept at task and language generalization. Nonetheless, they often fail when faced with unseen languages. In this work, we present LinguAlchemy, a regularization method that incorporates various linguistic information covering typological, geographical, and phylogenetic features to align PLMs representation to the corresponding linguistic information on each language. Our LinguAlchemy significantly improves the performance of mBERT and XLM-R on low-resource languages in multiple downstream tasks such as intent classification, news classification, and semantic relatedness compared to fully finetuned models and displaying a high degree of unseen language generalization. We further introduce AlchemyScale and AlchemyTune, extension of LinguAlchemy which adjusts the linguistic regularization weights automatically, alleviating the need for hyperparameter search.

Related papers

• Evaluating Morphological Compositional Generalization in Large Language Models [17.507983593566223]
  We investigate the morphological generalization abilities of large language models (LLMs) through the lens of compositionality. We focus on agglutinative languages such as Turkish and Finnish. Our analysis shows that LLMs struggle with morphological compositional generalization particularly when applied to novel word roots. While models can identify individual morphological combinations better than chance, their performance lacks systematicity, leading to significant accuracy gaps compared to humans.
  arXiv  Detail & Related papers  (2024-10-16T15:17:20Z)
• Interpretability of Language Models via Task Spaces [14.543168558734001]
  We present an alternative approach to interpret language models (LMs) We focus on the quality of LM processing, with a focus on their language abilities. We construct 'linguistic task spaces' that shed light on the connections LMs draw between language phenomena.
  arXiv  Detail & Related papers  (2024-06-10T16:34:30Z)
• Probing LLMs for Joint Encoding of Linguistic Categories [10.988109020181563]
  We propose a framework for testing the joint encoding of linguistic categories in Large Language Models (LLMs) We find evidence of joint encoding both at the same (related part-of-speech (POS) classes) and different (POS classes and related syntactic dependency relations) levels of linguistic hierarchy.
  arXiv  Detail & Related papers  (2023-10-28T12:46:40Z)
• Soft Language Clustering for Multilingual Model Pre-training [57.18058739931463]
  We propose XLM-P, which contextually retrieves prompts as flexible guidance for encoding instances conditionally. Our XLM-P enables (1) lightweight modeling of language-invariant and language-specific knowledge across languages, and (2) easy integration with other multilingual pre-training methods.
  arXiv  Detail & Related papers  (2023-06-13T08:08:08Z)
• Language Embeddings Sometimes Contain Typological Generalizations [0.0]
  We train neural models for a range of natural language processing tasks on a massively multilingual dataset of Bible translations in 1295 languages. The learned language representations are then compared to existing typological databases as well as to a novel set of quantitative syntactic and morphological features. We conclude that some generalizations are surprisingly close to traditional features from linguistic typology, but that most models, as well as those of previous work, do not appear to have made linguistically meaningful generalizations.
  arXiv  Detail & Related papers  (2023-01-19T15:09:59Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)
• Linguistic Typology Features from Text: Inferring the Sparse Features of World Atlas of Language Structures [73.06435180872293]
  We construct a recurrent neural network predictor based on byte embeddings and convolutional layers. We show that some features from various linguistic types can be predicted reliably.
  arXiv  Detail & Related papers  (2020-04-30T21:00:53Z)
• Bridging Linguistic Typology and Multilingual Machine Translation with Multi-View Language Representations [83.27475281544868]
  We use singular vector canonical correlation analysis to study what kind of information is induced from each source. We observe that our representations embed typology and strengthen correlations with language relationships. We then take advantage of our multi-view language vector space for multilingual machine translation, where we achieve competitive overall translation accuracy.
  arXiv  Detail & Related papers  (2020-04-30T16:25:39Z)
• Cross-lingual, Character-Level Neural Morphological Tagging [57.0020906265213]
  We train character-level recurrent neural taggers to predict morphological taggings for high-resource languages and low-resource languages together. Learning joint character representations among multiple related languages successfully enables knowledge transfer from the high-resource languages to the low-resource ones, improving accuracy by up to 30% over a monolingual model.
  arXiv  Detail & Related papers  (2017-08-30T08:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.