Multilingual Hallucination Gaps in Large Language Models

• URL: http://arxiv.org/abs/2410.18270v1
• Date: Wed, 23 Oct 2024 20:41:51 GMT
• Title: Multilingual Hallucination Gaps in Large Language Models
• Authors: Cléa Chataigner, Afaf Taïk, Golnoosh Farnadi,
• Abstract summary: We study the phenomenon of hallucinations across multiple languages in freeform text generation. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. Our results reveal variations in hallucination rates, especially between high and low resource languages.
• Score: 5.505634045241288
• License:
• Abstract: Large language models (LLMs) are increasingly used as alternatives to traditional search engines given their capacity to generate text that resembles human language. However, this shift is concerning, as LLMs often generate hallucinations, misleading or false information that appears highly credible. In this study, we explore the phenomenon of hallucinations across multiple languages in freeform text generation, focusing on what we call multilingual hallucination gaps. These gaps reflect differences in the frequency of hallucinated answers depending on the prompt and language used. To quantify such hallucinations, we used the FactScore metric and extended its framework to a multilingual setting. We conducted experiments using LLMs from the LLaMA, Qwen, and Aya families, generating biographies in 19 languages and comparing the results to Wikipedia pages. Our results reveal variations in hallucination rates, especially between high and low resource languages, raising important questions about LLM multilingual performance and the challenges in evaluating hallucinations in multilingual freeform text generation.

Related papers

• Mitigating Multilingual Hallucination in Large Vision-Language Models [35.75851356840673]
  We propose a two-stage Multilingual Hallucination Removal (MHR) framework for Large Vision-Language Models (LVLMs) Instead of relying on the intricate manual annotations of multilingual resources, we propose a novel cross-lingual alignment method. Our framework delivers an average increase of 19.0% in accuracy across 13 different languages.
  arXiv  Detail & Related papers  (2024-08-01T13:34:35Z)
• Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models [70.19081534515371]
  Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks. They generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. We propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers.
  arXiv  Detail & Related papers  (2024-07-04T18:47:42Z)
• Hallucination Diversity-Aware Active Learning for Text Summarization [46.00645048690819]
  Large Language Models (LLMs) have shown propensity to generate hallucinated outputs, i.e., texts that are factually incorrect or unsupported. Existing methods for alleviating hallucinations typically require costly human annotations to identify and correct hallucinations in LLM outputs. We propose the first active learning framework to alleviate LLM hallucinations, reducing costly human annotations of hallucination needed.
  arXiv  Detail & Related papers  (2024-04-02T02:30:27Z)
• Exploring and Evaluating Hallucinations in LLM-Powered Code Generation [14.438161741833687]
  Large Language Models (LLMs) produce outputs that deviate from users' intent, exhibit internal inconsistencies, or misalign with factual knowledge. Existing work mainly focuses on investing the hallucination in the domain of natural language generation. We conduct a thematic analysis of the LLM-generated code to summarize and categorize the hallucinations present in it. We propose HalluCode, a benchmark for evaluating the performance of code LLMs in recognizing hallucinations.
  arXiv  Detail & Related papers  (2024-04-01T07:31:45Z)
• Comparing Hallucination Detection Metrics for Multilingual Generation [62.97224994631494]
  This paper assesses how well various factual hallucination detection metrics identify hallucinations in generated biographical summaries across languages. We compare how well automatic metrics correlate to each other and whether they agree with human judgments of factuality. Our analysis reveals that while the lexical metrics are ineffective, NLI-based metrics perform well, correlating with human annotations in many settings and often outperforming supervised models.
  arXiv  Detail & Related papers  (2024-02-16T08:10:34Z)
• Hallucination Augmented Contrastive Learning for Multimodal Large Language Model [53.65682783591723]
  Multi-modal large language models (MLLMs) have been shown to efficiently integrate natural language with visual information to handle multi-modal tasks. However, MLLMs still face a fundamental limitation of hallucinations, where they tend to generate erroneous or fabricated information. In this paper, we address hallucinations in MLLMs from a novel perspective of representation learning.
  arXiv  Detail & Related papers  (2023-12-12T04:05:15Z)
• AutoHall: Automated Hallucination Dataset Generation for Large Language Models [56.92068213969036]
  This paper introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. We also propose a zero-resource and black-box hallucination detection method based on self-contradiction.
  arXiv  Detail & Related papers  (2023-09-30T05:20:02Z)
• HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models [146.87696738011712]
  Large language models (LLMs) are prone to generate hallucinations, i.e., content that conflicts with the source or cannot be verified by the factual knowledge. To understand what types of content and to which extent LLMs are apt to hallucinate, we introduce the Hallucination Evaluation benchmark for Large Language Models (HaluEval)
  arXiv  Detail & Related papers  (2023-05-19T15:36:27Z)
• Hallucinations in Large Multilingual Translation Models [70.10455226752015]
  Large-scale multilingual machine translation systems have demonstrated remarkable ability to translate directly between numerous languages. When deployed in the wild, these models may generate hallucinated translations which have the potential to severely undermine user trust and raise safety concerns. Existing research on hallucinations has primarily focused on small bilingual models trained on high-resource languages.
  arXiv  Detail & Related papers  (2023-03-28T16:17:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.