A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization

• URL: http://arxiv.org/abs/2608.08180v1
• Date: Sat, 08 Aug 2026 15:16:51 GMT
• Title: A Grounded and Decomposed Framework for Relation-Level Hallucination Evaluation in Abstractive Summarization
• Abstract summary: We present a refined and grounded framework for evaluating relation hallucination in abstractive summarization.<n>We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm.<n>We also introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models.
• Score: 2.6967769052209873
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Abstractive text summarization systems frequently generate fluent yet unfaithful summaries by fabricating or distorting relationships between entities and events. Such relation-level hallucinations undermine the reliability of generated summaries, particularly in high-stakes domains. In this work, we present a refined and grounded framework for evaluating relation hallucination in abstractive summarization. We present the empirical Relation Hallucination Index (RHI) by introducing a dependency-aware relation extraction algorithm that incorporates lemmatization-based normalization, named entity grounded subject resolution, passive agent recovery, negation-aware verb modeling, reporting verb filtering, nominal relation fallback, clausal propagation, and systematic deduplication. These enhancements improve the structural fidelity of extracted relation triples and reduce spurious matches during evaluation. In addition, we introduce a normalized formulation of RHI to ensure scale-invariant comparison between datasets and models. The revised metric decomposes hallucination into interpretable components, aggregates relation hallucination metric into a normalized relation faithfulness score. Extensive evaluation across multiple state-of-the-art summarization models demonstrates that the grounded extraction process yields more stable and discriminative hallucination measurements. The proposed framework advances automated relation-level faithfulness evaluation and supports coherence-aware, hallucination-sensitive model analysis.

Related papers

• Chain-based Adaptive Reconfiguration Over Lattices for Hallucination Reduction [0.0]
  CAROL is a probabilistic framework for test-time hallucination reduction in large language models.<n>CAROL defines a semantic uncertainty measure based on the consistency between generated responses and a trusted context.<n>CAROL unifies hallucination detection and mitigation within a single framework.
  arXiv  Detail & Related papers  (2026-05-26T21:28:51Z)
• K-Models: a Flexible and Interpretable Method for Ordinal Clustering with Application to Antigen-Antibody Interaction Profiles [65.21921642886367]
  K-Models is a novel framework that integrates ordinal constraints and estimates key underlying elements of the random process generating the observed functional profiles.<n>The proposed method is evaluated through simulations and real-world applications.
  arXiv  Detail & Related papers  (2026-05-14T13:35:44Z)
• Hallucination Benchmark for Speech Foundation Models [33.92968426403491]
  Hallucinations in automatic speech recognition (ASR) systems refer to fluent and coherent transcriptions produced by neural ASR models that are completely unrelated to the underlying acoustic input (i.e., the speech signal)<n>This apparent coherence can mislead subsequent processing stages and introduce serious risks, particularly in critical domains such as healthcare and law.<n>We introduce SHALLOW, the first benchmark framework that systematically categorizes and quantifies hallucination phenomena in ASR along four complementary axes: lexical, phonetic, morphological, and semantic.
  arXiv  Detail & Related papers  (2025-10-18T16:26:16Z)
• DEPTH: Hallucination-Free Relation Extraction via Dependency-Aware Sentence Simplification and Two-tiered Hierarchical Refinement [22.164114662885652]
  DEPTH is a framework that integrates Dependency-aware sEntence simPlification and Two-tiered Hierarchical refinement into the relation extraction pipeline.<n>We show that DEPTH reduces the average hallucination rate to 7.0% while achieving a 17.2% improvement in average F1 score over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-20T03:35:24Z)
• Reducing Hallucinations in Summarization via Reinforcement Learning with Entity Hallucination Index [2.2427832125073737]
  We introduce a rewarddriven fine-tuning framework to optimize for Entity Hallucination Index (EHI)<n>EHI is a metric designed to quantify the presence, correctness, and grounding of named entities in generated summaries.<n>Our approach does not rely on human-written factuality annotations, enabling scalable fine-tuning.
  arXiv  Detail & Related papers  (2025-07-30T15:00:00Z)
• Entity or Relation Embeddings? An Analysis of Encoding Strategies for Relation Extraction [19.019881161010474]
  Relation extraction is essentially a text classification problem, which can be tackled by fine-tuning a pre-trained language model (LM) Existing approaches therefore solve the problem in an indirect way: they fine-tune an LM to learn embeddings of the head and tail entities, and then predict the relationship from these entity embeddings. Our hypothesis in this paper is that relation extraction models can be improved by capturing relationships in a more direct way.
  arXiv  Detail & Related papers  (2023-12-18T09:58:19Z)
• AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation [57.8363998797433]
  We propose AMRFact, a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) Our approach parses factually consistent summaries into AMR graphs and injects controlled factual inconsistencies to create negative examples, allowing for coherent factually inconsistent summaries to be generated with high error-type coverage.
  arXiv  Detail & Related papers  (2023-11-16T02:56:29Z)
• Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
  We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
  arXiv  Detail & Related papers  (2023-08-24T16:06:36Z)
• Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis [89.04041100520881]
  This research proposes to retrieve textual and visual evidence based on the object, sentence, and whole image. We develop a novel approach to synthesize the object-level, image-level, and sentence-level information for better reasoning between the same and different modalities.
  arXiv  Detail & Related papers  (2023-05-25T15:26:13Z)
• HiURE: Hierarchical Exemplar Contrastive Learning for Unsupervised Relation Extraction [63.64877263289319]
  Unsupervised relation extraction aims to extract the relationship between entities from natural language sentences without prior information on relational scope or distribution.<n>We propose a novel contrastive learning framework named HiURE, which has the capability to derive hierarchical signals from relational feature space using cross hierarchy attention.<n> Experimental results on two public datasets demonstrate the advanced effectiveness and robustness of HiURE on unsupervised relation extraction when compared with state-of-the-art models.
  arXiv  Detail & Related papers  (2022-05-04T17:56:48Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.