Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?

• URL: http://arxiv.org/abs/2608.27115v1
• Date: Thu, 27 Aug 2026 13:27:18 GMT
• Title: Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations?
• Abstract summary: Cross-lingual alignment in multilingual language models is typically attributed to joint training.<n>We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training.
• Score: 25.211096690334355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cross-lingual alignment in multilingual language models is typically attributed to joint training: shared parameters, mixed-language batches, or explicit alignment objectives. We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training. By testing on strictly monolingual language models, such as the Goldfish model families and independently developed models from different research labs, we find three results. Correlation: these models develop alignable representational geometry across layers, with alignment strengthening as data scale, model scale, or linguistic proximity increases. Construction: a single Procrustes rotation fit on parallel sentences maps hidden states between models. Causation: the same rotation transfers functional content; patching a rotated English residual into a German model on a factual cloze flips the prediction to the donor's capital in most cases. We confirm that cross-lingual alignment can emerge from the structure of language and the information it carries rather than from joint training, and this points to practical future directions including model stitching, merging, and modular multilingual systems built from monolingual components.

Related papers

• Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations [25.913929585741034]
  Embedding alignment can mask real differences in how models internally represent a shared language.<n>We show that this assumption can be premature for decoder-only models.
  arXiv  Detail & Related papers  (2026-08-27T03:39:24Z)
• On the Limits of Model Merging for Multilinguality in Pre-Training [49.9160455376296]
  We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups.<n>We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference.
  arXiv  Detail & Related papers  (2026-05-25T13:38:33Z)
• One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging [30.967619128236777]
  We study weight-space merging for multilingual machine translation by fully fine-tuning language model.<n>Our experiments reveal that merging degrades performance, especially when target languages differ.<n>These findings suggest that multilingual fine-tuning may reshape geometry in ways that reduce compatibility with standard weight-space merging assumptions.
  arXiv  Detail & Related papers  (2026-04-03T08:45:26Z)
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments [57.273662221547056]
  In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. We observe that the existence of a predominant language during training boosts the performance of less frequent languages. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
  arXiv  Detail & Related papers  (2024-04-11T17:58:05Z)
• Improving In-context Learning of Multilingual Generative Language Models with Cross-lingual Alignment [42.624862172666624]
  We propose a simple yet effective cross-lingual alignment framework exploiting pairs of translation sentences. It aligns the internal sentence representations across different languages via multilingual contrastive learning. Experimental results show that even with less than 0.1 textperthousand of pre-training tokens, our alignment framework significantly boosts the cross-lingual abilities of generative language models.
  arXiv  Detail & Related papers  (2023-11-14T11:24:08Z)
• Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Models [28.036233760742125]
  We causally probe multilingual language models (XGLM and multilingual BERT) across various languages. We find significant neuron overlap across languages in autoregressive multilingual language models, but not masked language models.
  arXiv  Detail & Related papers  (2022-10-25T20:43:36Z)
• Language Anisotropic Cross-Lingual Model Editing [61.51863835749279]
  Existing work only studies the monolingual scenario, which lacks the cross-lingual transferability to perform editing simultaneously across languages. We propose a framework to naturally adapt monolingual model editing approaches to the cross-lingual scenario using parallel corpus. We empirically demonstrate the failure of monolingual baselines in propagating the edit to multiple languages and the effectiveness of the proposed language anisotropic model editing.
  arXiv  Detail & Related papers  (2022-05-25T11:38:12Z)
• Analyzing the Mono- and Cross-Lingual Pretraining Dynamics of Multilingual Language Models [73.11488464916668]
  This study investigates the dynamics of the multilingual pretraining process. We probe checkpoints taken from throughout XLM-R pretraining, using a suite of linguistic tasks. Our analysis shows that the model achieves high in-language performance early on, with lower-level linguistic skills acquired before more complex ones.
  arXiv  Detail & Related papers  (2022-05-24T03:35:00Z)
• Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining [3.4519649635864584]
  This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilingually more aligned representations.
  arXiv  Detail & Related papers  (2021-04-15T17:51:22Z)
• ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora [21.78571365050787]
  ERNIE-M is a new training method that encourages the model to align the representation of multiple languages with monolingual corpora. We generate pseudo-parallel sentences pairs on a monolingual corpus to enable the learning of semantic alignment between different languages. Experimental results show that ERNIE-M outperforms existing cross-lingual models and delivers new state-of-the-art results on various cross-lingual downstream tasks.
  arXiv  Detail & Related papers  (2020-12-31T15:52:27Z)
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling [73.40368222437912]
  We propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models to the unified multilingual model (student) Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.
  arXiv  Detail & Related papers  (2020-04-08T07:14:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.