Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility

• URL: http://arxiv.org/abs/2606.03291v1
• Date: Tue, 02 Jun 2026 07:55:52 GMT
• Title: Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility
• Abstract summary: We study multilingual unlearning by extending the TOFU benchmark to five languages.<n>We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable.
• Score: 27.073886293710945
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) can memorize sensitive facts, motivating unlearning methods that remove targeted knowledge without costly retraining. However, unlearning research remains heavily English-centric. We study multilingual unlearning by extending the TOFU benchmark to five languages, and fine-tune, unlearn, and query our models with different permutations of languages. We find that unlearning transfer, the ability of an unlearned model to "forget" facts in languages other than the unlearning language, is highly variable: e.g., it is strongest between languages sharing scripts and families, and we show that the unlearning language predicts which query languages are most likely to yield the strongest transfer. Layer-wise analysis reveals that unlearning leaves the shared cross-lingual latent space largely intact in early layers, instead operating primarily in later decoding layers. This suggests that unlearning does not truly erase knowledge, but rather induces superficial suppression. Exploiting this structure, a single inference-time steering direction reverses much of this suppression across languages, recovering 50% (Qwen) and 90% (Gemma) of the unlearned knowledge.

Related papers

• Layer-Targeted Multilingual Knowledge Erasure in Large Language Models [15.409568435026015]
  We identify intervention depth as the key factor determining multilingual generalization.<n>We propose MUTE, a framework that uses Centered Kernel Alignment (CKA) and Linguistic Regions Development Score (LRDS) to identify intermediate, language-agnostic layers.
  arXiv  Detail & Related papers  (2026-02-26T03:00:07Z)
• Evaluating Cross-Lingual Unlearning in Multilingual Language Models [7.530890774798437]
  Subspace-projection achieves strong cross-lingual forgetting with minimal degradation.<n>We show that multilingual forgetting depends on geometry in weight space, motivating subspace-based approaches for future unlearning systems.
  arXiv  Detail & Related papers  (2026-01-10T20:27:32Z)
• Multilingual Amnesia: On the Transferability of Unlearning in Multilingual LLMs [24.59074126514084]
  We study multilingual unlearning using the Aya-Expanse 8B model under two settings: data unlearning and concept unlearning.<n>We extend benchmarks for factual knowledge and stereotypes to ten languages through translation.<n>Our experiments show that unlearning in high-resource languages is generally more stable, with asymmetric transfer effects observed between typologically related languages.
  arXiv  Detail & Related papers  (2026-01-09T08:59:42Z)
• Tracing Multilingual Representations in LLMs with Cross-Layer Transcoders [51.380449540006985]
  Large Language Models (LLMs) can process many languages, yet how they internally represent this diversity remains unclear.<n>Do they form shared multilingual representations with language-specific decoding, and if so, why does performance still favor the dominant training language?<n>We analyze their internal mechanisms using cross-layer transcoders (CLT) and attribution graphs.
  arXiv  Detail & Related papers  (2025-11-13T22:51:06Z)
• Tracing Multilingual Factual Knowledge Acquisition in Pretraining [83.93508231653091]
  Large Language Models (LLMs) are capable of recalling multilingual factual knowledge present in their pretraining data.<n>We trace how factual recall and crosslingual consistency evolve during pretraining, focusing on OLMo-7B.<n>We find that both accuracy and consistency improve over time for most languages.
  arXiv  Detail & Related papers  (2025-05-20T18:39:56Z)
• MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing [78.62611800987817]
  Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. We propose a method called MoE-LPR (Mixture-of-Experts with Language Priors) to enhance the multilingual capability.
  arXiv  Detail & Related papers  (2024-08-21T07:43:49Z)
• Learn and Unlearn: Addressing Misinformation in Multilingual LLMs [11.179559502645697]
  This paper investigates the propagation of harmful information in multilingual large language models (LLMs)<n>Fake information, regardless of the language it is in, can spread across different languages, compromising the integrity and reliability of the generated content.<n>Standard unlearning techniques, which typically focus on English data, are insufficient in mitigating the spread of harmful content in multilingual contexts.
  arXiv  Detail & Related papers  (2024-06-19T18:01:08Z)
• Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models [38.10962690551031]
  Pretrained language models memorize vast amounts of information, including private and copyrighted data, raising significant safety concerns. Retraining these models after excluding sensitive data is prohibitively expensive, making machine unlearning a viable, cost-effective alternative. This paper presents a pioneering approach to machine unlearning for multilingual language models, selectively erasing information across different languages while maintaining overall performance.
  arXiv  Detail & Related papers  (2024-06-18T07:40:18Z)
• The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experiments [57.273662221547056]
  In this study, we investigate an unintuitive novel driver of cross-lingual generalisation: language imbalance. We observe that the existence of a predominant language during training boosts the performance of less frequent languages. As we extend our analysis to real languages, we find that infrequent languages still benefit from frequent ones, yet whether language imbalance causes cross-lingual generalisation there is not conclusive.
  arXiv  Detail & Related papers  (2024-04-11T17:58:05Z)
• How do Large Language Models Handle Multilingualism? [81.15060972112563]
  This study explores how large language models (LLMs) handle multilingualism. LLMs initially understand the query, converting multilingual inputs into English for task-solving. In the intermediate layers, they employ English for thinking and incorporate multilingual knowledge with self-attention and feed-forward structures.
  arXiv  Detail & Related papers  (2024-02-29T02:55:26Z)
• Language Representation Projection: Can We Transfer Factual Knowledge across Languages in Multilingual Language Models? [48.88328580373103]
  We propose two parameter-free $textbfL$anguage $textbfR$epresentation $textbfP$rojection modules (LRP2) The first module converts non-English representations into English-like equivalents, while the second module reverts English-like representations back into representations of the corresponding non-English language. Experimental results on the mLAMA dataset demonstrate that LRP2 significantly improves factual knowledge retrieval accuracy and facilitates knowledge transferability across diverse non-English languages.
  arXiv  Detail & Related papers  (2023-11-07T08:16:16Z)
• Towards continually learning new languages [66.36852845415916]
  batch-learning of languages can be economically beneficial, but the main challenge is catastrophic forgetting. We combine the qualities of weight factorization and elastic weight consolidation in order to counter catastrophic forgetting. We achieve 26 languages without catastrophic forgetting and a reasonable performance compared to training all languages from scratch.
  arXiv  Detail & Related papers  (2022-11-21T18:24:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.