MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation

• URL: http://arxiv.org/abs/2406.11566v1
• Date: Mon, 17 Jun 2024 14:03:50 GMT
• Title: MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation
• Authors: Jiakuan Xie, Pengfei Cao, Yuheng Chen, Yubo Chen, Kang Liu, Jun Zhao,
• Abstract summary: We focus on multilingual knowledge editing (MKE), which requires propagating updates across multiple languages. We introduce the Multilingual Knowledge Editing Benchmark (MKEB), a novel dataset comprising 12 languages. We also propose a method that enhances knowledge Editing with neuron-Masked Low-Rank Adaptation (MEMLA)
• Score: 18.087144677674786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge editing aims to adjust the knowledge within large language models (LLMs) to prevent their responses from becoming obsolete or inaccurate. However, existing works on knowledge editing are primarily conducted in a single language, which is inadequate for multilingual language models. In this paper, we focus on multilingual knowledge editing (MKE), which requires propagating updates across multiple languages. This necessity poses a significant challenge for the task. Furthermore, the limited availability of a comprehensive dataset for MKE exacerbates this challenge, hindering progress in this area. Hence, we introduce the Multilingual Knowledge Editing Benchmark (MKEB), a novel dataset comprising 12 languages and providing a complete evaluation framework. Additionally, we propose a method that enhances Multilingual knowledge Editing with neuron-Masked Low-Rank Adaptation (MEMLA). Specifically, we identify two categories of knowledge neurons to improve editing precision. Moreover, we perform LoRA-based editing with neuron masks to efficiently modify parameters and facilitate the propagation of updates across multiple languages. Experiments demonstrate that our method outperforms existing baselines and significantly enhances the multi-hop reasoning capability of the edited model, with minimal impact on its downstream task performance. The dataset and code will be made publicly available.

Related papers

• Cross-Lingual Multi-Hop Knowledge Editing -- Benchmarks, Analysis and a Simple Contrastive Learning based Approach [53.028586843468915]
  We propose the Cross-Lingual Multi-Hop Knowledge Editing paradigm, for measuring and analyzing the performance of various SoTA knowledge editing techniques in a cross-lingual setup. Specifically, we create a parallel cross-lingual benchmark, CROLIN-MQUAKE for measuring the knowledge editing capabilities. Following this, we propose a significantly improved system for cross-lingual multi-hop knowledge editing, CLEVER-CKE.
  arXiv  Detail & Related papers  (2024-07-14T17:18:16Z)
• Multilingual Knowledge Editing with Language-Agnostic Factual Neurons [98.73585104789217]
  We investigate how large language models (LLMs) represent multilingual factual knowledge. We find that the same factual knowledge in different languages generally activates a shared set of neurons, which we call language-agnostic factual neurons. Inspired by this finding, we propose a new MKE method by locating and modifying Language-Agnostic Factual Neurons (LAFN) to simultaneously edit multilingual knowledge.
  arXiv  Detail & Related papers  (2024-06-24T08:06:56Z)
• MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models [65.10456412127405]
  MLaKE is a benchmark for the adaptability of knowledge editing methods across five languages. MLaKE aggregates fact chains from Wikipedia across languages and generates questions in both free-form and multiple-choice. We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE.
  arXiv  Detail & Related papers  (2024-04-07T15:23:28Z)
• MPN: Leveraging Multilingual Patch Neuron for Cross-lingual Model Editing [10.81072864833299]
  We propose a simple yet effective method that trains multilingual patch neuron to store cross-lingual knowledge. It can be easily adapted to existing approaches to enhance their cross-lingual editing capabilities.
  arXiv  Detail & Related papers  (2024-01-06T10:40:24Z)
• Retrieval-augmented Multilingual Knowledge Editing [81.6690436581947]
  Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Knowledge editing (KE) has developed as an effective and economical alternative to inject new knowledge. We propose Retrieval-augmented Multilingual Knowledge Editor (ReMaKE) to update new knowledge in LLMs.
  arXiv  Detail & Related papers  (2023-12-20T14:08:58Z)
• Cross-Lingual Knowledge Editing in Large Language Models [73.12622532088564]
  Knowledge editing has been shown to adapt large language models to new knowledge without retraining from scratch. It is still unknown the effect of source language editing on a different target language. We first collect a large-scale cross-lingual synthetic dataset by translating ZsRE from English to Chinese.
  arXiv  Detail & Related papers  (2023-09-16T11:07:52Z)
• Adapters for Enhanced Modeling of Multilingual Knowledge and Text [54.02078328453149]
  Language models have been extended to multilingual language models (MLLMs) Knowledge graphs contain facts in an explicit triple format, which require careful curation and are only available in a few high-resource languages. We propose to enhance MLLMs with knowledge from multilingual knowledge graphs (MLKGs) so as to tackle language and knowledge graph tasks across many languages.
  arXiv  Detail & Related papers  (2022-10-24T21:33:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.