Multilingual Fine-Grained News Headline Hallucination Detection

• URL: http://arxiv.org/abs/2407.15975v1
• Date: Mon, 22 Jul 2024 18:37:53 GMT
• Title: Multilingual Fine-Grained News Headline Hallucination Detection
• Authors: Jiaming Shen, Tianqi Liu, Jialu Liu, Zhen Qin, Jay Pavagadhi, Simon Baumgartner, Michael Bendersky,
• Abstract summary: We introduce the first multilingual, fine-grained news headline hallucination detection dataset. This dataset contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance.
• Score: 40.62136051552646
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The popularity of automated news headline generation has surged with advancements in pre-trained language models. However, these models often suffer from the ``hallucination'' problem, where the generated headline is not fully supported by its source article. Efforts to address this issue have predominantly focused on English, using over-simplistic classification schemes that overlook nuanced hallucination types. In this study, we introduce the first multilingual, fine-grained news headline hallucination detection dataset that contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We conduct extensive experiments on this dataset under two settings. First, we implement several supervised fine-tuning approaches as preparatory solutions and demonstrate this dataset's challenges and utilities. Second, we test various large language models' in-context learning abilities and propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance in terms of the example-F1 metric. We release this dataset to foster further research in multilingual, fine-grained headline hallucination detection.

Related papers

• Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data [4.636499986218049]
  Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. We propose an approach to improve the sample efficiency of these models by creating corrupted grounding data.
  arXiv  Detail & Related papers  (2024-08-30T20:11:00Z)
• Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback [48.065569871444275]
  We propose detecting and mitigating hallucinations in Large Vision Language Models (LVLMs) via fine-grained AI feedback. We generate a small-size hallucination annotation dataset by proprietary models. Then, we propose a detect-then-rewrite pipeline to automatically construct preference dataset for training hallucination mitigating model.
  arXiv  Detail & Related papers  (2024-04-22T14:46:10Z)
• German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset [3.5206745486062636]
  This work presents absinth, a manually annotated dataset for hallucination detection in German news summarization. We open-source and release the absinth dataset to foster further research on hallucination detection in German.
  arXiv  Detail & Related papers  (2024-03-06T14:37:30Z)
• AutoHall: Automated Hallucination Dataset Generation for Large Language Models [56.92068213969036]
  This paper introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. We also propose a zero-resource and black-box hallucination detection method based on self-contradiction.
  arXiv  Detail & Related papers  (2023-09-30T05:20:02Z)
• Hallucinations in Large Multilingual Translation Models [70.10455226752015]
  Large-scale multilingual machine translation systems have demonstrated remarkable ability to translate directly between numerous languages. When deployed in the wild, these models may generate hallucinated translations which have the potential to severely undermine user trust and raise safety concerns. Existing research on hallucinations has primarily focused on small bilingual models trained on high-resource languages.
  arXiv  Detail & Related papers  (2023-03-28T16:17:59Z)
• Learning Cross-lingual Visual Speech Representations [108.68531445641769]
  Cross-lingual self-supervised visual representation learning has been a growing research topic in the last few years. We use the recently-proposed Raw Audio-Visual Speechs (RAVEn) framework to pre-train an audio-visual model with unlabelled data. Our experiments show that: (1) multi-lingual models with more data outperform monolingual ones, but, when keeping the amount of data fixed, monolingual models tend to reach better performance.
  arXiv  Detail & Related papers  (2023-03-14T17:05:08Z)
• "Why is this misleading?": Detecting News Headline Hallucinations with Explanations [30.52506534164537]
  We present a new framework named ExHalder to address hallucination detection. ExHalder adapts the knowledge from public natural language inference datasets into the news domain. It learns to generate natural language sentences to explain the hallucination detection results.
  arXiv  Detail & Related papers  (2023-02-12T04:21:49Z)
• Embedding Hallucination for Few-Shot Language Fine-tuning [14.244787327283335]
  We propose an Embedding Hallucination (EmbedHalluc) method, which generates auxiliary embedding-label pairs to expand the fine-tuning dataset. Experiments demonstrate that our proposed method is effective in a wide range of language tasks, outperforming current fine-tuning methods.
  arXiv  Detail & Related papers  (2022-05-03T04:55:50Z)
• Detecting Hallucinated Content in Conditional Neural Sequence Generation [165.68948078624499]
  We propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data.
  arXiv  Detail & Related papers  (2020-11-05T00:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.