Stress Testing Factual Consistency Metrics for Long-Document Summarization

• URL: http://arxiv.org/abs/2511.07689v1
• Date: Wed, 12 Nov 2025 01:11:49 GMT
• Title: Stress Testing Factual Consistency Metrics for Long-Document Summarization
• Authors: Zain Muhammad Mujahid, Dustin Wright, Isabelle Augenstein,
• Abstract summary: We systematically evaluate the reliability of six widely used reference-free factuality metrics, originally proposed for short-form summarization.<n>We probe metric robustness through seven factuality-preserving perturbations applied to summaries.<n>Our results reveal that existing short-form metrics produce inconsistent scores for semantically equivalent summaries and exhibit declining reliability for information-dense claims.
• Score: 36.761145124360944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating the factual consistency of abstractive text summarization remains a significant challenge, particularly for long documents, where conventional metrics struggle with input length limitations and long-range dependencies. In this work, we systematically evaluate the reliability of six widely used reference-free factuality metrics, originally proposed for short-form summarization, in the long-document setting. We probe metric robustness through seven factuality-preserving perturbations applied to summaries, namely paraphrasing, simplification, synonym replacement, logically equivalent negations, vocabulary reduction, compression, and source text insertion, and further analyze their sensitivity to retrieval context and claim information density. Across three long-form benchmark datasets spanning science fiction, legal, and scientific domains, our results reveal that existing short-form metrics produce inconsistent scores for semantically equivalent summaries and exhibit declining reliability for information-dense claims whose content is semantically similar to many parts of the source document. While expanding the retrieval context improves stability in some domains, no metric consistently maintains factual alignment under long-context conditions. Finally, our results highlight concrete directions for improving factuality evaluation, including multi-span reasoning, context-aware calibration, and training on meaning-preserving variations to enhance robustness in long-form summarization. We release all code, perturbed data, and scripts required to reproduce our results at https://github.com/zainmujahid/metricEval-longSum.

Related papers

• A Controllable Examination for Long-Context Language Models [62.845852724511964]
  This study introduces $textbfLongBioBench, a benchmark for evaluating long-context language models.<n>We show that most models still exhibit deficiencies in semantic understanding and elementary reasoning over retrieved results.<n>Our further analysis indicates some design choices employed by existing synthetic benchmarks, such as contextual non-coherence.
  arXiv  Detail & Related papers  (2025-06-03T14:23:06Z)
• Discourse-Driven Evaluation: Unveiling Factual Inconsistency in Long Document Summarization [7.218054628599005]
  We study factual inconsistency errors and connect them with a line of discourse analysis.<n>We propose a framework that decomposes long texts into discourse-inspired chunks.
  arXiv  Detail & Related papers  (2025-02-10T06:30:15Z)
• On Positional Bias of Faithfulness for Long-form Summarization [83.63283027830657]
  Large Language Models (LLMs) often exhibit positional bias in long-context settings, under-attending to information in the middle of inputs.<n>We investigate the presence of this bias in long-form summarization, its impact on faithfulness, and various techniques to mitigate this bias.
  arXiv  Detail & Related papers  (2024-10-31T03:50:15Z)
• FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction [85.26780391682894]
  We propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE) FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation.
  arXiv  Detail & Related papers  (2024-03-04T17:57:18Z)
• LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation [28.438103177230477]
  We evaluate the efficacy of automatic metrics for assessing the factual consistency of long document text summarisation. We propose a new evaluation framework, LongDocFACTScore, which is suitable for evaluating long document summarisation data sets.
  arXiv  Detail & Related papers  (2023-09-21T19:54:54Z)
• Evaluation of Faithfulness Using the Longest Supported Subsequence [52.27522262537075]
  We introduce a novel approach to evaluate faithfulness of machine-generated text by computing the longest noncontinuous of the claim that is supported by the context. Using a new human-annotated dataset, we finetune a model to generate Longest Supported Subsequence (LSS) Our proposed metric demonstrates an 18% enhancement over the prevailing state-of-the-art metric for faithfulness on our dataset.
  arXiv  Detail & Related papers  (2023-08-23T14:18:44Z)
• SMART: Sentences as Basic Units for Text Evaluation [48.5999587529085]
  In this paper, we introduce a new metric called SMART to mitigate such limitations. We treat sentences as basic units of matching instead of tokens, and use a sentence matching function to soft-match candidate and reference sentences. Our results show that system-level correlations of our proposed metric with a model-based matching function outperforms all competing metrics.
  arXiv  Detail & Related papers  (2022-08-01T17:58:05Z)
• Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation [42.63902468258758]
  We propose a novel metric to evaluate the factual consistency in text summarization via counterfactual estimation. We conduct a series of experiments on three public abstractive text summarization datasets.
  arXiv  Detail & Related papers  (2021-08-30T11:48:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.