BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning

• URL: http://arxiv.org/abs/2406.17764v1
• Date: Tue, 25 Jun 2024 17:48:56 GMT
• Title: BMIKE-53: Investigating Cross-Lingual Knowledge Editing with In-Context Learning
• Authors: Ercong Nie, Bo Shao, Zifeng Ding, Mingyang Wang, Helmut Schmid, Hinrich Schütze,
• Abstract summary: Knowledge editing (KE) has emerged as a viable solution for updating the knowledge of large language models. We introduce the BMIKE-53 benchmark for evaluating cross-lingual KE on 53 diverse languages across three KE task types. Our evaluation focuses on cross-lingual knowledge transfer in terms of reliability, generality, locality, and portability.
• Score: 43.059873703788206
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large language models (LLMs) possess extensive parametric knowledge, but this knowledge is difficult to update with new information because retraining is very expensive and infeasible for closed-source models. Knowledge editing (KE) has emerged as a viable solution for updating the knowledge of LLMs without compromising their overall performance. On-the-fly KE methods, inspired by in-context learning (ICL), have shown great promise and allow LLMs to be treated as black boxes. In the past, KE was primarily employed in English contexts, whereas the potential for cross-lingual KE in current English-centric LLMs has not been fully explored. To foster more research in this direction, we introduce the BMIKE-53 benchmark for evaluating cross-lingual KE on 53 diverse languages across three KE task types. We also propose a gradient-free KE method called Multilingual In-context Knowledge Editing (MIKE) and evaluate it on BMIKE-53. Our evaluation focuses on cross-lingual knowledge transfer in terms of reliability, generality, locality, and portability, offering valuable insights and a framework for future research in cross-lingual KE. Our code and data are publicly accessible via the anonymous repository at https://anonymous.4open.science/r/MIKE.

Related papers

• Cross-Lingual Multi-Hop Knowledge Editing [53.028586843468915]
  We propose the Cross-Lingual Multi-Hop Knowledge Editing paradigm, for measuring and analyzing the performance of various SoTA knowledge editing techniques in a cross-lingual setup. Specifically, we create a parallel cross-lingual benchmark, CROLIN-MQUAKE for measuring the knowledge editing capabilities. Following this, we propose a significantly improved system for cross-lingual multi-hop knowledge editing, CLEVER-CKE.
  arXiv  Detail & Related papers  (2024-07-14T17:18:16Z)
• Decipherment-Aware Multilingual Learning in Jointly Trained Language Models [2.8405971794435887]
  We show that the joint training methodology is a decipherment process pivotal for unsupervised multilingual learning. From an information-theoretic perspective, we demonstrate the importance of token alignment in challenging decipherment settings. Lastly, we apply lexical alignment to mBERT and investigate the contribution of aligning different lexicon groups to downstream performance.
  arXiv  Detail & Related papers  (2024-06-11T13:10:30Z)
• MLaKE: Multilingual Knowledge Editing Benchmark for Large Language Models [65.10456412127405]
  MLaKE is a benchmark for the adaptability of knowledge editing methods across five languages. MLaKE aggregates fact chains from Wikipedia across languages and generates questions in both free-form and multiple-choice. We evaluate the multilingual knowledge editing generalization capabilities of existing methods on MLaKE.
  arXiv  Detail & Related papers  (2024-04-07T15:23:28Z)
• Cross-Lingual Consistency of Factual Knowledge in Multilingual Language Models [2.6626950367610402]
  We study the cross-lingual consistency (CLC) of factual knowledge in various multilingual PLMs. We propose a Ranking-based Consistency (RankC) metric to evaluate knowledge consistency across languages independently from accuracy.
  arXiv  Detail & Related papers  (2023-10-16T13:19:17Z)
• Cross-Lingual Knowledge Editing in Large Language Models [73.12622532088564]
  Knowledge editing has been shown to adapt large language models to new knowledge without retraining from scratch. It is still unknown the effect of source language editing on a different target language. We first collect a large-scale cross-lingual synthetic dataset by translating ZsRE from English to Chinese.
  arXiv  Detail & Related papers  (2023-09-16T11:07:52Z)
• BUFFET: Benchmarking Large Language Models for Few-shot Cross-lingual Transfer [81.5984433881309]
  We introduce BUFFET, which unifies 15 diverse tasks across 54 languages in a sequence-to-sequence format. BUFFET is designed to establish a rigorous and equitable evaluation framework for few-shot cross-lingual transfer. Our findings reveal significant room for improvement in few-shot in-context cross-lingual transfer.
  arXiv  Detail & Related papers  (2023-05-24T08:06:33Z)
• The Impact of Cross-Lingual Adjustment of Contextual Word Representations on Zero-Shot Transfer [3.300216758849348]
  Large multilingual language models such as mBERT or XLM-R enable zero-shot cross-lingual transfer in various IR and NLP tasks. We propose a data- and compute-efficient method for cross-lingual adjustment of mBERT that uses a small parallel corpus to make embeddings of related words across languages similar to each other. We experiment with a typologically diverse set of languages (Spanish, Russian, Vietnamese, and Hindi) and extend their original implementations to new tasks. Our study reproduced gains in NLI for four languages, showed improved NER, XSR, and cross-lingual QA
  arXiv  Detail & Related papers  (2022-04-13T15:28:43Z)
• On the Language-specificity of Multilingual BERT and the Impact of Fine-tuning [7.493779672689531]
  The knowledge acquired by multilingual BERT (mBERT) has two components: a language-specific and a language-neutral one. This paper analyses the relationship between them, in the context of fine-tuning on two tasks.
  arXiv  Detail & Related papers  (2021-09-14T19:28:31Z)
• AM2iCo: Evaluating Word Meaning in Context across Low-ResourceLanguages with Adversarial Examples [51.048234591165155]
  We present AM2iCo, Adversarial and Multilingual Meaning in Context. It aims to faithfully assess the ability of state-of-the-art (SotA) representation models to understand the identity of word meaning in cross-lingual contexts. Results reveal that current SotA pretrained encoders substantially lag behind human performance.
  arXiv  Detail & Related papers  (2021-04-17T20:23:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.