Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations

• URL: http://arxiv.org/abs/2608.26576v1
• Date: Thu, 27 Aug 2026 03:39:24 GMT
• Title: Double Trouble: Bilingual Pretraining Leaves Language-Conditioned Effects in Shared-Language Representations
• Abstract summary: Embedding alignment can mask real differences in how models internally represent a shared language.<n>We show that this assumption can be premature for decoder-only models.
• Score: 25.913929585741034
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: When researchers compare multilingual models for probing, interpretability, or cross-lingual transfer, they often align embedding spaces and assume that shared-language representations are comparable. We show that this assumption can be premature for decoder-only models. We pretrain paired 310M-parameter models (one English-only, one bilingual) across eight typologically diverse languages, separately controlling for English exposure, total compute, and document overlap. After aligning on shared English vocabulary, we test held-out words and find that token embeddings look similar after alignment, but the deeper hidden states that the model uses for prediction do not. This gap holds for all eight languages and survives controls for document overlap and alternative alignment methods. This hidden-state mismatch grows through middle transformer layers, suggesting that it arises from contextual processing rather than the input representations where alignment is performed. Embedding alignment can mask real differences in how models internally represent a shared language, which matters for any downstream study that treats aligned models as interchangeable.

Related papers

• Cross-Lingual Alignment Without Joint Training: Do Monolingual Language Models Converge on Universal Representations? [25.211096690334355]
  Cross-lingual alignment in multilingual language models is typically attributed to joint training.<n>We ask whether monolingual models trained on non-parallel data learn alignable representations without joint training.
  arXiv  Detail & Related papers  (2026-08-27T13:27:18Z)
• Preferences for Idiomatic Language are Acquired Slowly -- and Forgotten Quickly: A Case Study on Swedish [0.6599344783327054]
  We investigate how language models develop preferences for textitidiomatic as compared to textitlinguistically acceptable Swedish.<n>For linguistic acceptability, we adapt existing benchmarks into a minimal-pair format.<n>Our findings suggest that idiomatic competence emerges more slowly than other linguistic abilities.
  arXiv  Detail & Related papers  (2026-02-03T12:57:39Z)
• False Friends Are Not Foes: Investigating Vocabulary Overlap in Multilingual Language Models [53.01170039144264]
  Subword tokenizers trained on multilingual corpora naturally produce overlapping tokens across languages.<n>Does token overlap facilitate cross-lingual transfer or instead introduce interference between languages?<n>We find that models with overlap outperform models with disjoint vocabularies.
  arXiv  Detail & Related papers  (2025-09-23T07:47:54Z)
• On the Acquisition of Shared Grammatical Representations in Bilingual Language Models [6.266732217239363]
  Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood.<n>We ask what happens to a monolingual language model when it begins to be trained on a second language.<n>To find evidence of shared multilingual representations, we turn to structural priming, a method used to study grammatical representations in humans.
  arXiv  Detail & Related papers  (2025-03-05T23:27:58Z)
• PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment [68.20851615263953]
  Large language models demonstrate reasonable multilingual abilities, despite predominantly English-centric pretraining. The spontaneous multilingual alignment in these models is shown to be weak, leading to unsatisfactory cross-lingual transfer and knowledge sharing. We propose PreAlign, a framework that establishes multilingual alignment prior to language model pretraining.
  arXiv  Detail & Related papers  (2024-07-23T06:59:53Z)
• Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Ability [11.000499414131324]
  We conduct experiments across four types of reasoning tasks. We find that the multilingual pretrained model does not always outperform an English-centric model. English appears to be a less suitable source language, and the choice of source language becomes less important when the English-centric model scales up.
  arXiv  Detail & Related papers  (2023-06-11T14:03:09Z)
• Language Model Tokenizers Introduce Unfairness Between Languages [98.92630681729518]
  We show how disparity in the treatment of different languages arises at the tokenization stage, well before a model is even invoked. Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs. We make the case that we should train future language models using multilingually fair subword tokenizers.
  arXiv  Detail & Related papers  (2023-05-17T14:17:57Z)
• A Crosslingual Investigation of Conceptualization in 1335 Languages [0.2216657815393579]
  We investigate differences in conceptualization across 1,335 languages by aligning concepts in a parallel corpus. We propose Conceptualizer, a method that creates a bipartite directed alignment graph between source language concepts and sets of target language strings. In a detailed linguistic analysis across all languages for one concept (bird') and an evaluation on gold standard data for 32 Swadesh concepts, we show that Conceptualizer has good alignment accuracy.
  arXiv  Detail & Related papers  (2023-05-15T09:27:34Z)
• M-SpeechCLIP: Leveraging Large-Scale, Pre-Trained Models for Multilingual Speech to Image Retrieval [56.49878599920353]
  This work investigates the use of large-scale, English-only pre-trained models (CLIP and HuBERT) for multilingual image-speech retrieval. For non-English image-speech retrieval, we outperform the current state-of-the-art performance by a wide margin both when training separate models for each language, and with a single model which processes speech in all three languages.
  arXiv  Detail & Related papers  (2022-11-02T14:54:45Z)
• MonoByte: A Pool of Monolingual Byte-level Language Models [4.491765479948667]
  We release 10 monolingual byte-level models rigorously pretrained under the same configuration. Because they are tokenizer-free, the problem of unseen token embeddings is eliminated. Experiments on QA and NLI tasks show that our monolingual models achieve competitive performance to the multilingual one.
  arXiv  Detail & Related papers  (2022-09-22T14:32:48Z)
• Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models [79.38278330678965]
  We find that common English pretraining corpora contain significant amounts of non-English text. This leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them.
  arXiv  Detail & Related papers  (2022-04-17T23:56:54Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• Improving Pretrained Cross-Lingual Language Models via Self-Labeled Word Alignment [49.45399359826453]
  Cross-lingual language models are typically pretrained with language modeling on multilingual text or parallel sentences. We introduce denoising word alignment as a new cross-lingual pre-training task. Experimental results show that our method improves cross-lingual transferability on various datasets.
  arXiv  Detail & Related papers  (2021-06-11T13:36:01Z)
• Bilingual alignment transfers to multilingual alignment for unsupervised parallel text mining [3.4519649635864584]
  This work presents methods for learning cross-lingual sentence representations using paired or unpaired bilingual texts. We hypothesize that the cross-lingual alignment strategy is transferable, and therefore a model trained to align only two languages can encode multilingually more aligned representations.
  arXiv  Detail & Related papers  (2021-04-15T17:51:22Z)
• Learning Contextualised Cross-lingual Word Embeddings and Alignments for Extremely Low-Resource Languages Using Parallel Corpora [63.5286019659504]
  We propose a new approach for learning contextualised cross-lingual word embeddings based on a small parallel corpus. Our method obtains word embeddings via an LSTM encoder-decoder model that simultaneously translates and reconstructs an input sentence.
  arXiv  Detail & Related papers  (2020-10-27T22:24:01Z)
• InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training [135.12061144759517]
  We present an information-theoretic framework that formulates cross-lingual language model pre-training. We propose a new pre-training task based on contrastive learning. By leveraging both monolingual and parallel corpora, we jointly train the pretext to improve the cross-lingual transferability of pre-trained models.
  arXiv  Detail & Related papers  (2020-07-15T16:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.