Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models

• URL: http://arxiv.org/abs/2603.29552v1
• Date: Tue, 31 Mar 2026 10:32:25 GMT
• Title: Bringing Up a Bilingual BabyLM: Investigating Multilingual Language Acquisition Using Small-Scale Models
• Abstract summary: We use language model training as a method for simulating a variety of highly controlled exposure conditions.<n>We create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation.<n>Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well.
• Score: 2.475536483276974
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multilingualism is incredibly common around the world, leading to many important theoretical and practical questions about how children learn multiple languages at once. For example, does multilingual acquisition lead to delays in learning? Are there better and worse ways to structure multilingual input? Many correlational studies address these questions, but it is surprisingly difficult to get definitive answers because children cannot be randomly assigned to be multilingual and data are typically not matched between languages. We use language model training as a method for simulating a variety of highly controlled exposure conditions, and create matched 100M-word mono- and bilingual datasets using synthetic data and machine translation. We train GPT-2 models on monolingual and bilingual data organized to reflect a range of exposure regimes, and evaluate their performance on perplexity, grammaticality, and semantic knowledge. Across model scales and measures, bilingual models perform similarly to monolingual models in one language, but show strong performance in the second language as well. These results suggest that there are no strong differences between different bilingual exposure regimes, and that bilingual input poses no in-principle challenges for agnostic statistical learners.

Related papers

• Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs [24.59074126514084]
  We study multilingual unlearning using the Aya-Expanse 8B model under two settings: data unlearning and concept unlearning.<n>We extend benchmarks for factual knowledge and stereotypes to ten languages through translation.<n>Our experiments show that unlearning in high-resource languages is generally more stable, with asymmetric transfer effects observed between typologically related languages.
  arXiv  Detail & Related papers  (2026-01-09T08:59:42Z)
• Revisiting Multilingual Data Mixtures in Language Model Pretraining [20.282622416939997]
  We study the impact of different multilingual data mixtures in pretraining large language models.<n>We find that combining English and multilingual data does not necessarily degrade the in-language performance of either group.<n>We do not observe a significant "curse of multilinguality" as the number of training languages increases.
  arXiv  Detail & Related papers  (2025-10-29T20:46:03Z)
• On the Acquisition of Shared Grammatical Representations in Bilingual Language Models [6.266732217239363]
  Crosslingual transfer is crucial to contemporary language models' multilingual capabilities, but how it occurs is not well understood.<n>We ask what happens to a monolingual language model when it begins to be trained on a second language.<n>To find evidence of shared multilingual representations, we turn to structural priming, a method used to study grammatical representations in humans.
  arXiv  Detail & Related papers  (2025-03-05T23:27:58Z)
• Learning Cross-lingual Visual Speech Representations [108.68531445641769]
  Cross-lingual self-supervised visual representation learning has been a growing research topic in the last few years. We use the recently-proposed Raw Audio-Visual Speechs (RAVEn) framework to pre-train an audio-visual model with unlabelled data. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance.
  arXiv  Detail & Related papers  (2023-03-14T17:05:08Z)
• Discovering Representation Sprachbund For Multilingual Pre-Training [139.05668687865688]
  We generate language representation from multilingual pre-trained models and conduct linguistic analysis. We cluster all the target languages into multiple groups and name each group as a representation sprachbund. Experiments are conducted on cross-lingual benchmarks and significant improvements are achieved compared to strong baselines.
  arXiv  Detail & Related papers  (2021-09-01T09:32:06Z)
• Are Multilingual Models Effective in Code-Switching? [57.78477547424949]
  We study the effectiveness of multilingual language models to understand their capability and adaptability to the mixed-language setting. Our findings suggest that pre-trained multilingual models do not necessarily guarantee high-quality representations on code-switching.
  arXiv  Detail & Related papers  (2021-03-24T16:20:02Z)
• How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models [96.32118305166412]
  We study a set of nine typologically diverse languages with readily available pretrained monolingual models on a set of five diverse monolingual downstream tasks. We find that languages which are adequately represented in the multilingual model's vocabulary exhibit negligible performance decreases over their monolingual counterparts.
  arXiv  Detail & Related papers  (2020-12-31T14:11:00Z)
• Structure-Level Knowledge Distillation For Multilingual Sequence Labeling [73.40368222437912]
  We propose to reduce the gap between monolingual models and the unified multilingual model by distilling the structural knowledge of several monolingual models to the unified multilingual model (student) Our experiments on 4 multilingual tasks with 25 datasets show that our approaches outperform several strong baselines and have stronger zero-shot generalizability than both the baseline model and teacher models.
  arXiv  Detail & Related papers  (2020-04-08T07:14:01Z)
• XPersona: Evaluating Multilingual Personalized Chatbot [76.00426517401894]
  We propose a multi-lingual extension of Persona-Chat, namely XPersona. Our dataset includes persona conversations in six different languages other than English for building and evaluating multilingual personalized agents.
  arXiv  Detail & Related papers  (2020-03-17T07:52:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.