Does Typological Blinding Impede Cross-Lingual Sharing?

• URL: http://arxiv.org/abs/2101.11888v1
• Date: Thu, 28 Jan 2021 09:32:08 GMT
• Title: Does Typological Blinding Impede Cross-Lingual Sharing?
• Authors: Johannes Bjerva and Isabelle Augenstein
• Abstract summary: We show that a model trained in a cross-lingual setting will pick up on typological cues from the input data. We investigate how cross-lingual sharing and performance is impacted.
• Score: 31.20201199491578
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bridging the performance gap between high- and low-resource languages has been the focus of much previous work. Typological features from databases such as the World Atlas of Language Structures (WALS) are a prime candidate for this, as such data exists even for very low-resource languages. However, previous work has only found minor benefits from using typological information. Our hypothesis is that a model trained in a cross-lingual setting will pick up on typological cues from the input data, thus overshadowing the utility of explicitly using such features. We verify this hypothesis by blinding a model to typological information, and investigate how cross-lingual sharing and performance is impacted. Our model is based on a cross-lingual architecture in which the latent weights governing the sharing between languages is learnt during training. We show that (i) preventing this model from exploiting typology severely reduces performance, while a control experiment reaffirms that (ii) encouraging sharing according to typology somewhat improves performance.

Related papers

• Language Model Pre-Training with Sparse Latent Typing [66.75786739499604]
  We propose a new pre-training objective, Sparse Latent Typing, which enables the model to sparsely extract sentence-level keywords with diverse latent types. Experimental results show that our model is able to learn interpretable latent type categories in a self-supervised manner without using any external knowledge.
  arXiv  Detail & Related papers  (2022-10-23T00:37:08Z)
• Cross-Lingual Fine-Grained Entity Typing [26.973783464706447]
  We present a unified cross-lingual fine-grained entity typing model capable of handling over 100 languages. We analyze this model's ability to generalize to languages and entities unseen during training.
  arXiv  Detail & Related papers  (2021-10-15T03:22:30Z)
• Towards Zero-shot Language Modeling [90.80124496312274]
  We construct a neural model that is inductively biased towards learning human languages. We infer this distribution from a sample of typologically diverse training languages. We harness additional language-specific side information as distant supervision for held-out languages.
  arXiv  Detail & Related papers  (2021-08-06T23:49:18Z)
• Unsupervised Domain Adaptation of a Pretrained Cross-Lingual Language Model [58.27176041092891]
  Recent research indicates that pretraining cross-lingual language models on large-scale unlabeled texts yields significant performance improvements. We propose a novel unsupervised feature decomposition method that can automatically extract domain-specific features from the entangled pretrained cross-lingual representations. Our proposed model leverages mutual information estimation to decompose the representations computed by a cross-lingual model into domain-invariant and domain-specific parts.
  arXiv  Detail & Related papers  (2020-11-23T16:00:42Z)
• Comparison of Interactive Knowledge Base Spelling Correction Models for Low-Resource Languages [81.90356787324481]
  Spelling normalization for low resource languages is a challenging task because the patterns are hard to predict. This work shows a comparison of a neural model and character language models with varying amounts on target language data. Our usage scenario is interactive correction with nearly zero amounts of training examples, improving models as more data is collected.
  arXiv  Detail & Related papers  (2020-10-20T17:31:07Z)
• SIGTYP 2020 Shared Task: Prediction of Typological Features [78.95376120154083]
  A major drawback hampering broader adoption of typological KBs is that they are sparsely populated. As typological features often correlate with one another, it is possible to predict them and thus automatically populate typological KBs. Overall, the task attracted 8 submissions from 5 teams, out of which the most successful methods make use of such feature correlations.
  arXiv  Detail & Related papers  (2020-10-16T08:47:24Z)
• Linguistic Typology Features from Text: Inferring the Sparse Features of World Atlas of Language Structures [73.06435180872293]
  We construct a recurrent neural network predictor based on byte embeddings and convolutional layers. We show that some features from various linguistic types can be predicted reliably.
  arXiv  Detail & Related papers  (2020-04-30T21:00:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.