Benchmarking Chinese Knowledge Rectification in Large Language Models

• URL: http://arxiv.org/abs/2409.05806v1
• Date: Mon, 9 Sep 2024 17:11:51 GMT
• Title: Benchmarking Chinese Knowledge Rectification in Large Language Models
• Authors: Tianhe Lu, Jizhan Fang, Yunzhi Yao, Xin Xu, Ningyu Zhang, Huajun Chen,
• Abstract summary: This paper introduces a benchmark for rectifying Chinese knowledge in Large Language Models via knowledge editing. We collect seven type of knowledge from various sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba. Through the analysis of this dataset, we uncover the challenges faced by current LLMs in mastering Chinese.
• Score: 43.9841600678381
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Language Models (LLMs) exhibit remarkable generative capabilities, they are not without flaws, particularly in the form of hallucinations. This issue is even more pronounced when LLMs are applied to specific languages and domains. For example, LLMs may generate nonsense information when handling Chinese ancient poetry, proverbs, or idioms, owing to the lack of specific knowledge. To this end, this paper introduces a benchmark for rectifying Chinese knowledge in LLMs via knowledge editing. Specifically, we introduce a new Chinese dataset, CKnowEdit, by collecting seven type of knowledge from various sources, including classical texts, idioms, and content from Baidu Tieba Ruozhiba, thereby accounting for the unique polyphony, antithesis, and logical constructs inherent in the Chinese language. Through the analysis of this dataset, we uncover the challenges faced by current LLMs in mastering Chinese. Furthermore, our evaluation of state-of-the-art knowledge editing techniques on this dataset unveil the substantial scope for advancement in the rectification of Chinese knowledge. Code and dataset are available at https://github.com/zjunlp/EasyEdit.

Related papers

• Cross-Lingual Multi-Hop Knowledge Editing -- Benchmarks, Analysis and a Simple Contrastive Learning based Approach [53.028586843468915]
  We propose the Cross-Lingual Multi-Hop Knowledge Editing paradigm, for measuring and analyzing the performance of various SoTA knowledge editing techniques in a cross-lingual setup. Specifically, we create a parallel cross-lingual benchmark, CROLIN-MQUAKE for measuring the knowledge editing capabilities. Following this, we propose a significantly improved system for cross-lingual multi-hop knowledge editing, CLEVER-CKE.
  arXiv  Detail & Related papers  (2024-07-14T17:18:16Z)
• Tracing the Roots of Facts in Multilingual Language Models: Independent, Shared, and Transferred Knowledge [16.923674220979]
  This study investigates how multilingual language models (ML-LMs) acquire and represent factual knowledge. We identify three patterns of acquiring and representing facts in ML-LMs: language-independent, cross-lingual shared and transferred. Our findings highlight the challenge of maintaining consistent factual knowledge across languages.
  arXiv  Detail & Related papers  (2024-03-08T10:09:57Z)
• Retrieval-augmented Multilingual Knowledge Editing [81.6690436581947]
  Knowledge represented in Large Language Models (LLMs) is quite often incorrect and can also become obsolete over time. Knowledge editing (KE) has developed as an effective and economical alternative to inject new knowledge. We propose Retrieval-augmented Multilingual Knowledge Editor (ReMaKE) to update new knowledge in LLMs.
  arXiv  Detail & Related papers  (2023-12-20T14:08:58Z)
• Cross-Lingual Knowledge Editing in Large Language Models [73.12622532088564]
  Knowledge editing has been shown to adapt large language models to new knowledge without retraining from scratch. It is still unknown the effect of source language editing on a different target language. We first collect a large-scale cross-lingual synthetic dataset by translating ZsRE from English to Chinese.
  arXiv  Detail & Related papers  (2023-09-16T11:07:52Z)
• Efficient and Effective Text Encoding for Chinese LLaMA and Alpaca [23.00353889531171]
  We propose a method to augment LLaMA with capabilities for understanding and generating Chinese text. We incorporate secondary pre-training using Chinese data and fine-tune the model with Chinese instruction datasets. Results on the C-Eval dataset yield competitive performance among the models with several times the size of ours.
  arXiv  Detail & Related papers  (2023-04-17T11:39:53Z)
• Adapters for Enhanced Modeling of Multilingual Knowledge and Text [54.02078328453149]
  Language models have been extended to multilingual language models (MLLMs) Knowledge graphs contain facts in an explicit triple format, which require careful curation and are only available in a few high-resource languages. We propose to enhance MLLMs with knowledge from multilingual knowledge graphs (MLKGs) so as to tackle language and knowledge graph tasks across many languages.
  arXiv  Detail & Related papers  (2022-10-24T21:33:42Z)
• Intrinsic Knowledge Evaluation on Chinese Language Models [5.293979881130493]
  This paper proposes four tasks on syntactic, semantic, commonsense, and factual knowledge, aggregating to a total of $39,308$ questions. Our probes and knowledge data prove to be a reliable benchmark for evaluating pre-trained Chinese LMs.
  arXiv  Detail & Related papers  (2020-11-29T04:34:39Z)
• X-FACTR: Multilingual Factual Knowledge Retrieval from Pretrained Language Models [103.75890012041366]
  Language models (LMs) have proven surprisingly successful at capturing factual knowledge. However, studies on LMs' factual representation ability have almost invariably been performed on English. We create a benchmark of cloze-style probes for 23 typologically diverse languages.
  arXiv  Detail & Related papers  (2020-10-13T05:29:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.