An In-Vitro Study on Cross-Lingual Generalization in Language Models

• URL: http://arxiv.org/abs/2605.26683v1
• Date: Tue, 26 May 2026 08:20:20 GMT
• Title: An In-Vitro Study on Cross-Lingual Generalization in Language Models
• Abstract summary: Cross-lingual transfer in language models is difficult to study because lexical overlap, morphology, data imbalance, and tokenization are entangled.<n>We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure.<n>We find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure.
• Score: 1.9875440739965626
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cross-lingual transfer in language models is difficult to study in natural corpora because lexical overlap, morphology, data imbalance, and tokenization are entangled. We introduce an in-vitro framework with two procedurally generated languages that share the same ontology, typed grammar, and compositional structure, but differ in surface realization. This lets us independently vary lexical distance, minority-language proportion, tokenizer training regime, and vocabulary size, while evaluating transfer on a masked minority-language condition whose lexical forms are never observed during training. Across 700 controlled runs, we find that transfer is governed less by tokenizer balance or raw lexical similarity than by whether tokenization preserves reusable cross-lingual substructure. Smaller vocabularies often improve masked transfer by keeping words decomposable into shared fragments, whereas larger vocabularies can turn forms into language-specific atoms. We further show that transfer emerges as a staged process: grammatical and type-level competence precede masked lexical generalization. Finally, we attempt to explain this mechanism through tokenizer bridges and show that bridge strength correlates strongly with masked reachability.

Related papers

• Shared Circuits for Shared Grammar: Tracing Subject-Verb Agreement Across Languages [18.33287987422947]
  We find that languages with overt person/number inflection exhibit more similar agreement circuitry than non-conjugating languages.<n>Many implicated heads display similar attention patterns across languages, suggesting that cross-lingual overlap reflects shared functional roles as well as shared localization.
  arXiv  Detail & Related papers  (2026-08-19T05:05:15Z)
• When Meanings Meet: Investigating the Emergence and Quality of Shared Concept Spaces during Multilingual Language Model Training [57.230355403478995]
  We investigate the development of language-agnostic concept spaces during pretraining of EuroLLM.<n>We find that shared concept spaces emerge early and continue to refine, but that alignment with them is language-dependent.<n>In contrast to prior work, our fine-grained manual analysis reveals that some apparent gains in translation quality reflect shifts in behavior.
  arXiv  Detail & Related papers  (2026-01-30T11:23:01Z)
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• On the Acquisition of Shared Grammatical Representations in Bilingual Language Models [6.266732217239363]
  Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood.<n>We ask what happens to a monolingual language model when it begins to be trained on a second language.<n>To find evidence of shared multilingual representations, we turn to structural priming, a method used to study grammatical representations in humans.
  arXiv  Detail & Related papers  (2025-03-05T23:27:58Z)
• Cross-Linguistic Syntactic Difference in Multilingual BERT: How Good is It and How Does It Affect Transfer? [50.48082721476612]
  Multilingual BERT (mBERT) has demonstrated considerable cross-lingual syntactic ability. We investigate the distributions of grammatical relations induced from mBERT in the context of 24 typologically different languages.
  arXiv  Detail & Related papers  (2022-12-21T09:44:08Z)
• Cross-Lingual Ability of Multilingual Masked Language Models: A Study of Language Structure [54.01613740115601]
  We study three language properties: constituent order, composition and word co-occurrence. Our main conclusion is that the contribution of constituent order and word co-occurrence is limited, while the composition is more crucial to the success of cross-linguistic transfer.
  arXiv  Detail & Related papers  (2022-03-16T07:09:35Z)
• Oolong: Investigating What Makes Transfer Learning Hard with Controlled Studies [21.350999136803843]
  We systematically transform the language of the GLUE benchmark, altering one axis of crosslingual variation at a time. We find that models can largely recover from syntactic-style shifts, but cannot recover from vocabulary misalignment. Our experiments provide insights into the factors of cross-lingual transfer that researchers should most focus on when designing language transfer scenarios.
  arXiv  Detail & Related papers  (2022-02-24T19:00:39Z)
• VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generation [77.82373082024934]
  We plug a cross-attention module into the Transformer encoder to explicitly build the interdependence between languages. It can effectively avoid the degeneration of predicting masked words only conditioned on the context in its own language. The proposed cross-lingual model delivers new state-of-the-art results on various cross-lingual understanding tasks of the XTREME benchmark.
  arXiv  Detail & Related papers  (2020-10-30T03:41:38Z)
• Translation Artifacts in Cross-lingual Transfer Learning [51.66536640084888]
  We show that machine translation can introduce subtle artifacts that have a notable impact in existing cross-lingual models. In natural language inference, translating the premise and the hypothesis independently can reduce the lexical overlap between them. We also improve the state-of-the-art in XNLI for the translate-test and zero-shot approaches by 4.3 and 2.8 points, respectively.
  arXiv  Detail & Related papers  (2020-04-09T17:54:30Z)
• Understanding Cross-Lingual Syntactic Transfer in Multilingual Recurrent Neural Networks [3.9342247746757435]
  It is now established that modern neural language models can be successfully trained on multiple languages simultaneously. But what kind of knowledge is really shared among languages within these models? In this paper we dissect different forms of cross-lingual transfer and look for its most determining factors. We find that exposing our LMs to a related language does not always increase grammatical knowledge in the target language, and that optimal conditions for lexical-semantic transfer may not be optimal for syntactic transfer.
  arXiv  Detail & Related papers  (2020-03-31T09:48:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.