Vocabulary shapes cross-lingual variation of word-order learnability in language models

• URL: http://arxiv.org/abs/2603.19427v1
• Date: Thu, 19 Mar 2026 19:42:21 GMT
• Title: Vocabulary shapes cross-lingual variation of word-order learnability in language models
• Abstract summary: We study why some languages like Czech permit free word order, while others like English do not.<n>We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability.<n>A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages does not explain cross-lingual variation.<n>The structure of the word and subword vocabulary strongly predicts the model surprisal.
• Score: 6.944578798815993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Why do some languages like Czech permit free word order, while others like English do not? We address this question by pretraining transformer language models on a spectrum of synthetic word-order variants of natural languages. We observe that greater word-order irregularity consistently raises model surprisal, indicating reduced learnability. Sentence reversal, however, affects learnability only weakly. A coarse distinction of free- (e.g., Czech and Finnish) and fixed-word-order languages (e.g., English and French) does not explain cross-lingual variation. Instead, the structure of the word and subword vocabulary strongly predicts the model surprisal. Overall, vocabulary structure emerges as a key driver of computational word-order learnability across languages.

Related papers

• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languages [55.157295899188476]
  neural machine translation systems learn to map sentences of different languages into a common representation space. In this work, we test this hypothesis by zero-shot translating from unseen languages. We demonstrate that this setup enables zero-shot translation from entirely unseen languages.
  arXiv  Detail & Related papers  (2024-08-05T07:58:58Z)
• Crosslinguistic word order variation reflects evolutionary pressures of dependency and information locality [4.869029215261254]
  About 40% of the world's languages have subject-verb-object order, and about 40% have subject-object-verb order. We show that variation in word order reflects different ways of balancing competing pressures of dependency locality and information locality. Our findings suggest that syntactic structure and usage across languages co-adapt to support efficient communication under limited cognitive resources.
  arXiv  Detail & Related papers  (2022-06-09T02:56:53Z)
• Handling Compounding in Mobile Keyboard Input [7.309321705635677]
  This paper proposes a framework to improve the typing experience of mobile users in morphologically rich languages. Smartphone keyboards typically support features such as input decoding, corrections and predictions that all rely on language models. We show that this method brings around 20% word error rate reduction in a variety of compounding languages.
  arXiv  Detail & Related papers  (2022-01-17T15:28:58Z)
• Allocating Large Vocabulary Capacity for Cross-lingual Language Model Pre-training [59.571632468137075]
  We find that many languages are under-represented in recent cross-lingual language models due to the limited vocabulary capacity. We propose an algorithm VoCap to determine the desired vocabulary capacity of each language. In order to address the issues, we propose k-NN-based target sampling to accelerate the expensive softmax.
  arXiv  Detail & Related papers  (2021-09-15T14:04:16Z)
• Revisiting Language Encoding in Learning Multilingual Representations [70.01772581545103]
  We propose a new approach called Cross-lingual Language Projection (XLP) to replace language embedding. XLP projects the word embeddings into language-specific semantic space, and then the projected embeddings will be fed into the Transformer model. Experiments show that XLP can freely and significantly boost the model performance on extensive multilingual benchmark datasets.
  arXiv  Detail & Related papers  (2021-02-16T18:47:10Z)
• On the Evolution of Word Order [7.2610922684683645]
  We show that an optimal language is one with fixed word order. We also show that adding information to the sentence, such as case markers and noun-verb distinction, reduces the need for fixed word order.
  arXiv  Detail & Related papers  (2021-01-23T20:30:17Z)
• Improving Multilingual Models with Language-Clustered Vocabularies [8.587129426070979]
  We introduce a novel procedure for multilingual vocabulary generation that combines the separately trained vocabularies of several automatically derived language clusters. Our experiments show improvements across languages on key multilingual benchmark tasks.
  arXiv  Detail & Related papers  (2020-10-24T04:49:15Z)
• Investigating Cross-Linguistic Adjective Ordering Tendencies with a Latent-Variable Model [66.84264870118723]
  We present the first purely corpus-driven model of multi-lingual adjective ordering in the form of a latent-variable model. We provide strong converging evidence for the existence of universal, cross-linguistic, hierarchical adjective ordering tendencies.
  arXiv  Detail & Related papers  (2020-10-09T18:27:55Z)
• On the Importance of Word Order Information in Cross-lingual Sequence Labeling [80.65425412067464]
  Cross-lingual models that fit into the word order of the source language might fail to handle target languages. We investigate whether making models insensitive to the word order of the source language can improve the adaptation performance in target languages.
  arXiv  Detail & Related papers  (2020-01-30T03:35:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.