Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models

• URL: http://arxiv.org/abs/2606.00284v1
• Date: Fri, 29 May 2026 19:18:20 GMT
• Title: Parameter Alignment Mitigates Catastrophic Forgetting in Multilingual Expert Language Models
• Abstract summary: Organizing training around language families reduces cross-language interference but cannot alone prevent forgetting of the general knowledge needed for downstream tasks.<n>We present a suite of five layer-aware parameter alignment strategies: hard layer freezing, soft regularization, post-hoc weight reversion, and model merging.<n>We systematically compare our alignment strategies against two unregularized CPT baselines on benchmarks spanning 32 training languages from five language families, plus held-out languages.
• Score: 11.557479638024674
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While continual pretraining~(CPT) is a practical way to extend large language models to new languages, naïve finetuning on targeted data erodes existing capabilities through catastrophic forgetting. Organizing training around language families reduces cross-language interference but cannot alone prevent forgetting of the general knowledge needed for downstream tasks. We link this forgetting to parameter drift in multilingual CPT and present a suite of five layer-aware parameter alignment strategies: hard layer freezing, soft regularization, post-hoc weight reversion, and model merging. We systematically compare our alignment strategies against two unregularized CPT baselines on benchmarks spanning 32 training languages from five language families, plus held-out languages, across four evaluation axes: perplexity, reading comprehension, physical reasoning, and translation. Parameter alignment substantially reduces forgetting at minimal cost to language acquisition: layer freezing and regularization best preserve comprehension, whereas post-hoc reversion yields the strongest translation gains. Together, these results map the acquisition--forgetting frontier for family-expert CPT and offer practical deployment guidelines pairing each strategy to the tasks it best serves.

Related papers

• A Data-Efficient Path to Multilingual LLMs: Language Expansion via Post-training PARAM$Δ$ Integration into Upcycled MoE [61.0405668388095]
  We introduce method, which upcycles a dense model into a Mixture-of-Experts(MoE) architecture, allocating different experts to different languages.<n>Experiments demonstrate method's superiority even against baselines with similar FLOPs or number of parameters.
  arXiv  Detail & Related papers  (2026-05-18T08:59:08Z)
• Cross-Lingual Jailbreak Detection via Semantic Codebooks [0.0]
  Safety mechanisms for large language models (LLMs) remain predominantly English-centric, creating systematic vulnerabilities in multilingual deployment.<n>We investigate whether such attacks can be mitigated through language-agnostic semantic similarity without retraining or language-specific adaptation.
  arXiv  Detail & Related papers  (2026-04-28T14:43:40Z)
• Adaptive Constraint Propagation: Scaling Structured Inference for Large Language Models via Meta-Reinforcement Learning [6.908972852063454]
  MetaJuLS learns universal constraint propagation policies applicable across languages and tasks without task-specific retraining.<n>On Universal Dependencies across 10 languages, MetaJuLS adapts to new languages and tasks with 5--10 gradient steps (5--15 seconds) rather than requiring hours of task-specific training.<n>Mechanistic analysis reveals the policy discovers human-like parsing strategies (easy-first) and novel non-intuitives.
  arXiv  Detail & Related papers  (2025-12-31T20:03:49Z)
• Language steering in latent space to mitigate unintended code-switching [1.1330938617817454]
  Large Language Models (LLMs) often exhibit unintended code-switching, reducing reliability in downstream tasks.<n>We propose latent-space language steering, a lightweight inference-time method that identifies language directions via PCA on parallel translations.<n>Our approach mitigates code-switching while preserving semantics with negligible computational overhead.
  arXiv  Detail & Related papers  (2025-10-11T19:49:38Z)
• MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safety [56.77103365251923]
  Large Language Models (LLMs) are susceptible to adversarial attacks such as jailbreaking.<n>This vulnerability is exacerbated in multilingual settings, where multilingual safety-aligned data is often limited.<n>We introduce a multilingual guardrail with reasoning for prompt classification.
  arXiv  Detail & Related papers  (2025-04-21T17:15:06Z)
• InstructionCP: A fast approach to transfer Large Language Models into target language [55.2480439325792]
  InsCP integrates instruction tags into the CP process to prevent loss of conversational proficiency while acquiring new languages. Our experiments demonstrate that InsCP retains conversational and Reinforcement Learning from Human Feedback abilities. This approach requires only 0.1 billion tokens of high-quality instruction-following data, thereby reducing resource consumption.
  arXiv  Detail & Related papers  (2024-05-30T15:45:13Z)
• No Train but Gain: Language Arithmetic for training-free Language Adapters enhancement [59.37775534633868]
  We introduce a novel method called language arithmetic, which enables training-free post-processing. The effectiveness of the proposed solution is demonstrated on three downstream tasks in a MAD-X-based set of cross-lingual schemes.
  arXiv  Detail & Related papers  (2024-04-24T08:52:40Z)
• Measuring Catastrophic Forgetting in Cross-Lingual Transfer Paradigms: Exploring Tuning Strategies [4.118037156777793]
  Cross-lingual transfer is a promising technique to solve tasks in less-resourced languages.<n>We compare two fine-tuning approaches combined with zero-shot and full-shot learning approaches for large language models.
  arXiv  Detail & Related papers  (2023-09-12T09:37:08Z)
• Language Agnostic Multilingual Information Retrieval with Contrastive Learning [59.26316111760971]
  We present an effective method to train multilingual information retrieval systems. We leverage parallel and non-parallel corpora to improve the pretrained multilingual language models. Our model can work well even with a small number of parallel sentences.
  arXiv  Detail & Related papers  (2022-10-12T23:53:50Z)
• Continual Learning in Multilingual NMT via Language-Specific Embeddings [92.91823064720232]
  It consists in replacing the shared vocabulary with a small language-specific vocabulary and fine-tuning the new embeddings on the new language's parallel data. Because the parameters of the original model are not modified, its performance on the initial languages does not degrade.
  arXiv  Detail & Related papers  (2021-10-20T10:38:57Z)
• Towards Lifelong Learning of Multilingual Text-To-Speech Synthesis [87.75833205560406]
  This work presents a lifelong learning approach to train a multilingual Text-To-Speech (TTS) system. It does not require pooled data from all languages altogether, and thus alleviates the storage and computation burden.
  arXiv  Detail & Related papers  (2021-10-09T07:00:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.