Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation

• URL: http://arxiv.org/abs/2604.03141v1
• Date: Fri, 03 Apr 2026 16:03:56 GMT
• Title: Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation
• Abstract summary: We evaluate the factuality of long-form output generated by large language models (LLMs)<n>We propose a comprehensive factuality evaluation framework that jointly measures precision and recall.<n>Our analysis reveals that current LLMs perform substantially better on precision than on recall.
• Score: 32.04613130489821
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating the factuality of long-form output generated by large language models (LLMs) remains challenging, particularly when responses are open-ended and contain many fine-grained factual statements. Existing evaluation methods primarily focus on precision: they decompose a response into atomic claims and verify each claim against external knowledge sources such as Wikipedia. However, this overlooks an equally important dimension of factuality: recall, whether the generated response covers the relevant facts that should be included. We propose a comprehensive factuality evaluation framework that jointly measures precision and recall. Our method leverages external knowledge sources to construct reference facts and determine whether they are captured in generated text. We further introduce an importance-aware weighting scheme based on relevance and salience. Our analysis reveals that current LLMs perform substantially better on precision than on recall, suggesting that factual incompleteness remains a major limitation of long-form generation and that models are generally better at covering highly important facts than the full set of relevant facts.

Related papers

• Towards Comprehensive Stage-wise Benchmarking of Large Language Models in Fact-Checking [64.97768177044355]
  Large Language Models (LLMs) are increasingly deployed in real-world fact-checking systems.<n>We present FactArena, a fully automated arena-style evaluation framework.<n>Our analyses reveal significant discrepancies between static claim-verification accuracy and end-to-end fact-checking competence.
  arXiv  Detail & Related papers  (2026-01-06T02:51:56Z)
• Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models [2.0861090421004937]
  Large Language Models (LLMs) are trained on vast and diverse internet corpora that often include inaccurate or misleading content.<n>This review systematically analyzes how LLM-generated content is evaluated for factual accuracy.
  arXiv  Detail & Related papers  (2025-08-05T19:20:05Z)
• RelationalFactQA: A Benchmark for Evaluating Tabular Fact Retrieval from Large Language Models [9.211266032947497]
  We demonstrate that fact retrieval is substantially more difficult than isolated point-wise queries.<n>Our experiments reveal that even stateofthe-art LLMs struggle significantly, not exceeding 25% factual accuracy.<n>These findings underscore limitations in current LLMs' ability to synthesize structured factual knowledge.
  arXiv  Detail & Related papers  (2025-05-27T16:33:38Z)
• VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts [13.694698772833627]
  We introduce VeriFact, a factuality evaluation framework designed to enhance fact extraction.<n>We also introduce FactRBench, a benchmark that evaluates both precision and recall in long-form model responses.<n> Empirical evaluations show that VeriFact significantly enhances fact completeness and preserves complex facts with critical relational information.
  arXiv  Detail & Related papers  (2025-05-14T18:02:37Z)
• Exploring the Generalizability of Factual Hallucination Mitigation via Enhancing Precise Knowledge Utilization [49.95746521480879]
  We introduce PKUE (Precise Knowledge Utilization Enhancement), which fine-tunes the model on self-generated responses to precise and simple factual questions.<n>Extensive experiments demonstrate that PKUE significantly improves LLM overall performance.
  arXiv  Detail & Related papers  (2025-02-26T13:34:52Z)
• FactReasoner: A Probabilistic Approach to Long-Form Factuality Assessment for Large Language Models [59.171510592986735]
  We propose FactReasoner, a new factuality assessor that relies on probabilistic reasoning to assess the factuality of a long-form generated response.<n>Our experiments on labeled and unlabeled benchmark datasets demonstrate clearly that FactReasoner improves considerably over state-of-the-art prompt-based approaches.
  arXiv  Detail & Related papers  (2025-02-25T19:01:48Z)
• Improving Factuality with Explicit Working Memory [68.39261790277615]
  Large language models can generate factually inaccurate content, a problem known as hallucination.<n>We introduce EWE (Explicit Working Memory), a novel approach that enhances factuality in long-form text generation by integrating a working memory that receives real-time feedback from external resources.
  arXiv  Detail & Related papers  (2024-12-24T00:55:59Z)
• FactAlign: Long-form Factuality Alignment of Large Language Models [35.067998820937284]
  Large language models have demonstrated significant potential as the next-generation information access engines. We propose FactAlign, a novel alignment framework designed to enhance the factuality of long-form responses. Our experiments on open-domain prompts and information-seeking questions demonstrate that FactAlign significantly improves the factual accuracy of LLM responses.
  arXiv  Detail & Related papers  (2024-10-02T16:03:13Z)
• FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction [85.26780391682894]
  We propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE) FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation.
  arXiv  Detail & Related papers  (2024-03-04T17:57:18Z)
• Merging Facts, Crafting Fallacies: Evaluating the Contradictory Nature of Aggregated Factual Claims in Long-Form Generations [63.90357081534995]
  Long-form generations from large language models (LLMs) contain a mix of factual and non-factual claims. We show that strong open-source models like Llama-chat can generate paragraphs that contain verifiable facts, but the facts are combined into a non-factual paragraph due to entity ambiguity. We introduce an enhanced metric, D-FActScore, specifically designed for content with ambiguous entities.
  arXiv  Detail & Related papers  (2024-02-08T12:36:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.