Related papers: German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset

German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset

URL: http://arxiv.org/abs/2403.03750v2
Date: Thu, 14 Mar 2024 12:30:54 GMT
Title: German also Hallucinates! Inconsistency Detection in News Summaries with the Absinth Dataset
Authors: Laura Mascarell, Ribin Chalumattu, Annette Rios,
Abstract summary: This work presents absinth, a manually annotated dataset for hallucination detection in German news summarization. We open-source and release the absinth dataset to foster further research on hallucination detection in German.
Score: 3.5206745486062636
License: http://creativecommons.org/licenses/by-nc-sa/4.0/
Abstract: The advent of Large Language Models (LLMs) has led to remarkable progress on a wide range of natural language processing tasks. Despite the advances, these large-sized models still suffer from hallucinating information in their output, which poses a major issue in automatic text summarization, as we must guarantee that the generated summary is consistent with the content of the source document. Previous research addresses the challenging task of detecting hallucinations in the output (i.e. inconsistency detection) in order to evaluate the faithfulness of the generated summaries. However, these works primarily focus on English and recent multilingual approaches lack German data. This work presents absinth, a manually annotated dataset for hallucination detection in German news summarization and explores the capabilities of novel open-source LLMs on this task in both fine-tuning and in-context learning settings. We open-source and release the absinth dataset to foster further research on hallucination detection in German.

Related papers

Multilingual Fine-Grained News Headline Hallucination Detection [40.62136051552646]
We introduce the first multilingual, fine-grained news headline hallucination detection dataset. This dataset contains over 11 thousand pairs in 5 languages, each annotated with detailed hallucination types by experts. We propose two novel techniques, language-dependent demonstration selection and coarse-to-fine prompting, to boost the few-shot hallucination detection performance.
arXiv Detail & Related papers (2024-07-22T18:37:53Z)
Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs [70.15262704746378]
We propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (10% Rouge-L) in terms of producing coherent summaries.
arXiv Detail & Related papers (2024-07-05T20:25:04Z)
SEMQA: Semi-Extractive Multi-Source Question Answering [94.04430035121136]
We introduce a new QA task for answering multi-answer questions by summarizing multiple diverse sources in a semi-extractive fashion. We create the first dataset of this kind, QuoteSum, with human-written semi-extractive answers to natural and generated questions.
arXiv Detail & Related papers (2023-11-08T18:46:32Z)
AutoHall: Automated Hallucination Dataset Generation for Large Language Models [56.92068213969036]
This paper introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. We also propose a zero-resource and black-box hallucination detection method based on self-contradiction.
arXiv Detail & Related papers (2023-09-30T05:20:02Z)
Hallucination Reduction in Long Input Text Summarization [2.6745438139282283]
Hallucination in text summarization poses significant obstacles to the accuracy and reliability of the generated summaries. We have incorporated the techniques of data filtering and joint entity and summary generation (JAENS) in the fine-tuning of the Longformer-Decoder (LED) model. Our experiments show that the fine-tuned LED model performs well in generating the paper abstract.
arXiv Detail & Related papers (2023-09-28T18:22:16Z)
Detecting and Mitigating Hallucinations in Multilingual Summarisation [40.5267502712576]
Hallucinations pose a significant challenge to the reliability of neural models for abstractive summarisation. We develop a novel metric, mFACT, evaluating the faithfulness of non-English summaries. We then propose a simple but effective method to reduce hallucinations with a cross-lingual transfer.
arXiv Detail & Related papers (2023-05-23T02:59:25Z)
mFACE: Multilingual Summarization with Factual Consistency Evaluation [79.60172087719356]
Abstractive summarization has enjoyed renewed interest in recent years, thanks to pre-trained language models and the availability of large-scale datasets. Despite promising results, current models still suffer from generating factually inconsistent summaries. We leverage factual consistency evaluation models to improve multilingual summarization.
arXiv Detail & Related papers (2022-12-20T19:52:41Z)
Survey of Hallucination in Natural Language Generation [69.9926849848132]
Natural Language Generation (NLG) has improved exponentially in recent years thanks to the development of sequence-to-sequence deep learning technologies. Deep learning based generation is prone to hallucinate unintended text, which degrades the system performance. This survey serves to facilitate collaborative efforts among researchers in tackling the challenge of hallucinated texts in NLG.
arXiv Detail & Related papers (2022-02-08T03:55:01Z)
Detecting Hallucinated Content in Conditional Neural Sequence Generation [165.68948078624499]
We propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data.
arXiv Detail & Related papers (2020-11-05T00:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.