Every Language Counts: Learn and Unlearn in Multilingual LLMs

• URL: http://arxiv.org/abs/2406.13748v1
• Date: Wed, 19 Jun 2024 18:01:08 GMT
• Title: Every Language Counts: Learn and Unlearn in Multilingual LLMs
• Authors: Taiming Lu, Philipp Koehn,
• Abstract summary: This paper investigates the propagation of harmful information in multilingual large language models (LLMs) Fake information, regardless of the language it is in, can spread across different languages, compromising the integrity and reliability of the generated content. Standard unlearning techniques, which typically focus on English data, are insufficient in mitigating the spread of harmful content in multilingual contexts.
• Score: 11.42788038138136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper investigates the propagation of harmful information in multilingual large language models (LLMs) and evaluates the efficacy of various unlearning methods. We demonstrate that fake information, regardless of the language it is in, once introduced into these models through training data, can spread across different languages, compromising the integrity and reliability of the generated content. Our findings reveal that standard unlearning techniques, which typically focus on English data, are insufficient in mitigating the spread of harmful content in multilingual contexts and could inadvertently reinforce harmful content across languages. We show that only by addressing harmful responses in both English and the original language of the harmful data can we effectively eliminate generations for all languages. This underscores the critical need for comprehensive unlearning strategies that consider the multilingual nature of modern LLMs to enhance their safety and reliability across diverse linguistic landscapes.

Related papers

• Lens: Rethinking Multilingual Enhancement for Large Language Models [70.85065197789639]
  Lens is a novel approach to enhance multilingual capabilities of large language models (LLMs) It operates by manipulating the hidden representations within the language-agnostic and language-specific subspaces from top layers of LLMs. It achieves superior results with much fewer computational resources compared to existing post-training approaches.
  arXiv  Detail & Related papers  (2024-10-06T08:51:30Z)
• Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Models [62.91524967852552]
  Large language models (LLMs) are typically multilingual due to pretraining on diverse multilingual corpora. But can these models relate corresponding concepts across languages, effectively being crosslingual? This study evaluates six state-of-the-art LLMs on inherently crosslingual tasks.
  arXiv  Detail & Related papers  (2024-06-23T15:15:17Z)
• Teaching LLMs to Abstain across Languages via Multilingual Feedback [40.84205285309612]
  We show that multilingual feedback helps identify knowledge gaps across diverse languages, cultures, and communities. Extensive experiments demonstrate that our multilingual feedback approach outperforms various strong baselines. Further analysis reveals that multilingual feedback is both an effective and a more equitable abstain strategy to serve diverse language speakers.
  arXiv  Detail & Related papers  (2024-06-22T21:59:12Z)
• Towards Truthful Multilingual Large Language Models: Benchmarking and Alignment Strategies [38.3269908062146]
  We construct a benchmark for truthfulness evaluation in multilingual scenarios. We propose Fact-aware Multilingual Selective Synergy (FaMSS) to optimize the data allocation across a large number of languages.
  arXiv  Detail & Related papers  (2024-06-20T15:59:07Z)
• Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Models [38.10962690551031]
  Pretrained language models memorize vast amounts of information, including private and copyrighted data, raising significant safety concerns. Retraining these models after excluding sensitive data is prohibitively expensive, making machine unlearning a viable, cost-effective alternative. This paper presents a pioneering approach to machine unlearning for multilingual language models, selectively erasing information across different languages while maintaining overall performance.
  arXiv  Detail & Related papers  (2024-06-18T07:40:18Z)
• From One to Many: Expanding the Scope of Toxicity Mitigation in Language Models [10.807067327137855]
  As language models embrace multilingual capabilities, it's crucial our safety measures keep pace. In the absence of sufficient annotated datasets across languages, we employ translated data to evaluate and enhance our mitigation techniques. This allows us to examine the effects of translation quality and the cross-lingual transfer on toxicity mitigation.
  arXiv  Detail & Related papers  (2024-03-06T17:51:43Z)
• Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages [60.162717568496355]
  Large language models (LLMs) have been pre-trained on multilingual corpora. Their performance still lags behind in most languages compared to a few resource-rich languages.
  arXiv  Detail & Related papers  (2024-02-19T15:07:32Z)
• UltraLink: An Open-Source Knowledge-Enhanced Multilingual Supervised Fine-tuning Dataset [69.33424532827608]
  Open-source large language models (LLMs) have gained significant strength across diverse fields. In this work, we construct an open-source multilingual supervised fine-tuning dataset. The resulting UltraLink dataset comprises approximately 1 million samples across five languages.
  arXiv  Detail & Related papers  (2024-02-07T05:05:53Z)
• Cross-lingual Machine Reading Comprehension with Language Branch Knowledge Distillation [105.41167108465085]
  Cross-lingual Machine Reading (CLMRC) remains a challenging problem due to the lack of large-scale datasets in low-source languages. We propose a novel augmentation approach named Language Branch Machine Reading (LBMRC) LBMRC trains multiple machine reading comprehension (MRC) models proficient in individual language. We devise a multilingual distillation approach to amalgamate knowledge from multiple language branch models to a single model for all target languages.
  arXiv  Detail & Related papers  (2020-10-27T13:12:17Z)
• On Negative Interference in Multilingual Models: Findings and A Meta-Learning Treatment [59.995385574274785]
  We show that, contrary to previous belief, negative interference also impacts low-resource languages. We present a meta-learning algorithm that obtains better cross-lingual transferability and alleviates negative interference.
  arXiv  Detail & Related papers  (2020-10-06T20:48:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.