Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation

• URL: http://arxiv.org/abs/2506.00288v2
• Date: Tue, 03 Jun 2025 10:17:34 GMT
• Title: Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
• Authors: Ahmed Elhady, Eneko Agirre, Mikel Artetxe,
• Abstract summary: We show that including English does not impact validation perplexity, yet it is critical for the emergence of downstream capabilities in the target language.<n>We introduce curriculum learning and exponential moving average (EMA) of weights as effective alternatives to mitigate the need for English.
• Score: 39.97090144259728
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continued pretraining (CPT) is a popular approach to adapt existing large language models (LLMs) to new languages. When doing so, it is common practice to include a portion of English data in the mixture, but its role has not been carefully studied to date. In this work, we show that including English does not impact validation perplexity, yet it is critical for the emergence of downstream capabilities in the target language. We introduce a language-agnostic benchmark for in-context learning (ICL), which reveals catastrophic forgetting early on CPT when English is not included. This in turn damages the ability of the model to generalize to downstream prompts in the target language as measured by perplexity, even if it does not manifest in terms of accuracy until later in training, and can be tied to a big shift in the model parameters. Based on these insights, we introduce curriculum learning and exponential moving average (EMA) of weights as effective alternatives to mitigate the need for English. All in all, our work sheds light into the dynamics by which emergent abilities arise when doing CPT for language adaptation, and can serve as a foundation to design more effective methods in the future.

Related papers

• Cross-Lingual Transfer of Debiasing and Detoxification in Multilingual LLMs: An Extensive Investigation [6.781972039785424]
  Recent generative large language models (LLMs) show remarkable performance in non-English languages.<n>When prompted in those languages they tend to express higher harmful social biases and toxicity levels.<n>We investigate the impact of different finetuning methods on the model's bias and toxicity, but also on its ability to produce fluent and diverse text.
  arXiv  Detail & Related papers  (2024-12-18T17:05:08Z)
• PreAlign: Boosting Cross-Lingual Transfer by Early Establishment of Multilingual Alignment [68.20851615263953]
  Large language models demonstrate reasonable multilingual abilities, despite predominantly English-centric pretraining. The spontaneous multilingual alignment in these models is shown to be weak, leading to unsatisfactory cross-lingual transfer and knowledge sharing. We propose PreAlign, a framework that establishes multilingual alignment prior to language model pretraining.
  arXiv  Detail & Related papers  (2024-07-23T06:59:53Z)
• InstructionCP: A fast approach to transfer Large Language Models into target language [55.2480439325792]
  InsCP integrates instruction tags into the CP process to prevent loss of conversational proficiency while acquiring new languages. Our experiments demonstrate that InsCP retains conversational and Reinforcement Learning from Human Feedback abilities. This approach requires only 0.1 billion tokens of high-quality instruction-following data, thereby reducing resource consumption.
  arXiv  Detail & Related papers  (2024-05-30T15:45:13Z)
• Efficiently Adapting Pretrained Language Models To New Languages [9.33333013114014]
  Recent large language models (LLM) exhibit sub-optimal performance on low-resource languages. We study how to efficiently adapt any existing pretrained LLM to a new language without running into these issues.
  arXiv  Detail & Related papers  (2023-11-09T20:59:08Z)
• Improving Language Plasticity via Pretraining with Active Forgetting [63.36484652568976]
  We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Experiments with RoBERTa show that models pretrained with our forgetting mechanism demonstrate faster convergence during language adaptation.
  arXiv  Detail & Related papers  (2023-07-03T17:12:44Z)
• Pre-Trained Language-Meaning Models for Multilingual Parsing and Generation [14.309869321407522]
  We introduce multilingual pre-trained language-meaning models based on Discourse Representation Structures (DRSs) Since DRSs are language neutral, cross-lingual transfer learning is adopted to further improve the performance of non-English tasks. automatic evaluation results show that our approach achieves the best performance on both the multilingual DRS parsing and DRS-to-text generation tasks.
  arXiv  Detail & Related papers  (2023-05-31T19:00:33Z)
• Language Contamination Explains the Cross-lingual Capabilities of English Pretrained Models [79.38278330678965]
  We find that common English pretraining corpora contain significant amounts of non-English text. This leads to hundreds of millions of foreign language tokens in large-scale datasets. We then demonstrate that even these small percentages of non-English data facilitate cross-lingual transfer for models trained on them.
  arXiv  Detail & Related papers  (2022-04-17T23:56:54Z)
• Language Models are Few-shot Multilingual Learners [66.11011385895195]
  We evaluate the multilingual skills of the GPT and T5 models in conducting multi-class classification on non-English languages. We show that, given a few English examples as context, pre-trained language models can predict not only English test samples but also non-English ones.
  arXiv  Detail & Related papers  (2021-09-16T03:08:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.