Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation

• URL: http://arxiv.org/abs/2411.05641v1
• Date: Fri, 08 Nov 2024 15:35:43 GMT
• Title: Evaluating Large Language Model Capability in Vietnamese Fact-Checking Data Generation
• Authors: Long Truong To, Hung Tuan Le, Dat Van-Thanh Nguyen, Manh Trong Nguyen, Tri Thien Nguyen, Tin Van Huynh, Kiet Van Nguyen,
• Abstract summary: Large Language Models (LLMs) are being applied to a range of complex language tasks. This paper explores the use of LLMs for automatic data generation for the Vietnamese fact-checking task. We develop an automatic data construction process using simple prompt techniques and explore several methods to improve the quality of the generated data.
• Score: 1.0173628293062005
• License:
• Abstract: Large Language Models (LLMs), with gradually improving reading comprehension and reasoning capabilities, are being applied to a range of complex language tasks, including the automatic generation of language data for various purposes. However, research on applying LLMs for automatic data generation in low-resource languages like Vietnamese is still underdeveloped and lacks comprehensive evaluation. In this paper, we explore the use of LLMs for automatic data generation for the Vietnamese fact-checking task, which faces significant data limitations. Specifically, we focus on fact-checking data where claims are synthesized from multiple evidence sentences to assess the information synthesis capabilities of LLMs. We develop an automatic data construction process using simple prompt techniques on LLMs and explore several methods to improve the quality of the generated data. To evaluate the quality of the data generated by LLMs, we conduct both manual quality assessments and performance evaluations using language models. Experimental results and manual evaluations illustrate that while the quality of the generated data has significantly improved through fine-tuning techniques, LLMs still cannot match the data quality produced by humans.

Related papers

• Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning [71.2981957820888]
  We propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets. The framework initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. The generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality.
  arXiv  Detail & Related papers  (2024-11-21T02:30:53Z)
• SELF-GUIDE: Better Task-Specific Instruction Following via Self-Synthetic Finetuning [70.21358720599821]
  Large language models (LLMs) hold the promise of solving diverse tasks when provided with appropriate natural language prompts. We propose SELF-GUIDE, a multi-stage mechanism in which we synthesize task-specific input-output pairs from the student LLM. We report an absolute improvement of approximately 15% for classification tasks and 18% for generation tasks in the benchmark's metrics.
  arXiv  Detail & Related papers  (2024-07-16T04:41:58Z)
• CLAIM Your Data: Enhancing Imputation Accuracy with Contextual Large Language Models [0.18416014644193068]
  This paper introduces the Contextual Language model for Accurate Imputation Method (CLAIM) Unlike traditional imputation methods, CLAIM utilizes contextually relevant natural language descriptors to fill missing values. Our evaluations across diverse datasets and missingness patterns reveal CLAIM's superior performance over existing imputation techniques.
  arXiv  Detail & Related papers  (2024-05-28T00:08:29Z)
• Crossing Linguistic Horizons: Finetuning and Comprehensive Evaluation of Vietnamese Large Language Models [11.563813473794013]
  Open-sourced large language models (LLMs) exhibit limited effectiveness in processing Vietnamese. To mitigate these issues, we have finetuned LLMs specifically for Vietnamese. Our evaluation results reveal that the fine-tuned LLMs exhibit enhanced comprehension and generative capabilities in Vietnamese.
  arXiv  Detail & Related papers  (2024-03-05T07:13:28Z)
• Exploring Precision and Recall to assess the quality and diversity of LLMs [82.21278402856079]
  We introduce a novel evaluation framework for Large Language Models (LLMs) such as textscLlama-2 and textscMistral. This approach allows for a nuanced assessment of the quality and diversity of generated text without the need for aligned corpora.
  arXiv  Detail & Related papers  (2024-02-16T13:53:26Z)
• Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning [79.32236399694077]
  Low-quality data in the training set are usually detrimental to instruction tuning. We propose a novel method, termed "reflection-tuning" This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data.
  arXiv  Detail & Related papers  (2023-10-18T05:13:47Z)
• CulturaX: A Cleaned, Enormous, and Multilingual Dataset for Large Language Models in 167 Languages [86.90220551111096]
  Training datasets for large language models (LLMs) are often not fully disclosed. We present CulturaX, a substantial multilingual dataset with 6.3 trillion tokens in 167 languages.
  arXiv  Detail & Related papers  (2023-09-17T23:49:10Z)
• From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning [52.257422715393574]
  We introduce a self-guided methodology for Large Language Models (LLMs) to autonomously discern and select cherry samples from open-source datasets. Our key innovation, the Instruction-Following Difficulty (IFD) metric, emerges as a pivotal metric to identify discrepancies between a model's expected responses and its intrinsic generation capability.
  arXiv  Detail & Related papers  (2023-08-23T09:45:29Z)
• CARE-MI: Chinese Benchmark for Misinformation Evaluation in Maternity and Infant Care [14.326936563564171]
  We present a benchmark, CARE-MI, for evaluating misinformation in large language models (LLMs) Our proposed benchmark fills the gap between the extensive usage of LLMs and the lack of datasets for assessing the misinformation generated by these models. Using our benchmark, we conduct extensive experiments and found that current Chinese LLMs are far from perfect in the topic of maternity and infant care.
  arXiv  Detail & Related papers  (2023-07-04T03:34:19Z)
• MEGA: Multilingual Evaluation of Generative AI [23.109803506475174]
  Generative AI models have shown impressive performance on many Natural Language Processing tasks. Most studies on generative LLMs have been restricted to English. It is unclear how capable these models are at understanding and generating text in other languages.
  arXiv  Detail & Related papers  (2023-03-22T13:03:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.