TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain

• URL: http://arxiv.org/abs/2508.13798v1
• Date: Tue, 19 Aug 2025 12:57:45 GMT
• Title: TracSum: A New Benchmark for Aspect-Based Summarization with Sentence-Level Traceability in Medical Domain
• Authors: Bohao Chu, Meijie Li, Sameh Frihat, Chengyu Gu, Georg Lodde, Elisabeth Livingstone, Norbert Fuhr,
• Abstract summary: We introduce TracSum, a novel benchmark for traceable, aspect-based summarization.<n> generated summaries are paired with sentence-level citations, enabling users to trace back to the original context.<n>We show that TracSum can serve as an effective benchmark for traceable, aspect-based summarization tasks.
• Score: 1.5732353205551508
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While document summarization with LLMs has enhanced access to textual information, concerns about the factual accuracy of these summaries persist, especially in the medical domain. Tracing evidence from which summaries are derived enables users to assess their accuracy, thereby alleviating this concern. In this paper, we introduce TracSum, a novel benchmark for traceable, aspect-based summarization, in which generated summaries are paired with sentence-level citations, enabling users to trace back to the original context. First, we annotate 500 medical abstracts for seven key medical aspects, yielding 3.5K summary-citation pairs. We then propose a fine-grained evaluation framework for this new task, designed to assess the completeness and consistency of generated content using four metrics. Finally, we introduce a summarization pipeline, Track-Then-Sum, which serves as a baseline method for comparison. In experiments, we evaluate both this baseline and a set of LLMs on TracSum, and conduct a human evaluation to assess the evaluation results. The findings demonstrate that TracSum can serve as an effective benchmark for traceable, aspect-based summarization tasks. We also observe that explicitly performing sentence-level tracking prior to summarization enhances generation accuracy, while incorporating the full context further improves completeness.

Related papers

• Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs [70.15262704746378]
  We propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (10% Rouge-L) in terms of producing coherent summaries.
  arXiv  Detail & Related papers  (2024-07-05T20:25:04Z)
• Attribute Structuring Improves LLM-Based Evaluation of Clinical Text Summaries [56.31117605097345]
  Large language models (LLMs) have shown the potential to generate accurate clinical text summaries, but still struggle with issues regarding grounding and evaluation.<n>Here, we explore a general mitigation framework using Attribute Structuring (AS), which structures the summary evaluation process.<n>AS consistently improves the correspondence between human annotations and automated metrics in clinical text summarization.
  arXiv  Detail & Related papers  (2024-03-01T21:59:03Z)
• On Context Utilization in Summarization with Large Language Models [83.84459732796302]
  Large language models (LLMs) excel in abstractive summarization tasks, delivering fluent and pertinent summaries. Recent advancements have extended their capabilities to handle long-input contexts, exceeding 100k tokens. We conduct the first comprehensive study on context utilization and position bias in summarization.
  arXiv  Detail & Related papers  (2023-10-16T16:45:12Z)
• SummIt: Iterative Text Summarization via ChatGPT [12.966825834765814]
  We propose SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback. We also conduct a human evaluation to validate the effectiveness of the iterative refinements and identify a potential issue of over-correction.
  arXiv  Detail & Related papers  (2023-05-24T07:40:06Z)
• NapSS: Paragraph-level Medical Text Simplification via Narrative Prompting and Sentence-matching Summarization [46.772517928718216]
  We propose a summarize-then-simplify two-stage strategy, which we call NapSS. NapSS identifies the relevant content to simplify while ensuring that the original narrative flow is preserved. Our model achieves significantly better than the seq2seq baseline on an English medical corpus.
  arXiv  Detail & Related papers  (2023-02-11T02:20:25Z)
• Towards Interpretable Summary Evaluation via Allocation of Contextual Embeddings to Reference Text Topics [1.5749416770494706]
  The multifaceted interpretable summary evaluation method (MISEM) is based on allocation of a summary's contextual token embeddings to semantic topics identified in the reference text. MISEM achieves a promising.404 Pearson correlation with human judgment on the TAC'08 dataset.
  arXiv  Detail & Related papers  (2022-10-25T17:09:08Z)
• A Training-free and Reference-free Summarization Evaluation Metric via Centrality-weighted Relevance and Self-referenced Redundancy [60.419107377879925]
  We propose a training-free and reference-free summarization evaluation metric. Our metric consists of a centrality-weighted relevance score and a self-referenced redundancy score. Our methods can significantly outperform existing methods on both multi-document and single-document summarization evaluation.
  arXiv  Detail & Related papers  (2021-06-26T05:11:27Z)
• Towards Clinical Encounter Summarization: Learning to Compose Discharge Summaries from Prior Notes [15.689048077818324]
  This paper introduces the task of generating discharge summaries for a clinical encounter. We introduce two new measures, faithfulness and hallucination rate for evaluation. Results across seven medical sections and five models show that a summarization architecture that supports traceability yields promising results.
  arXiv  Detail & Related papers  (2021-04-27T22:45:54Z)
• Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries [74.28810048824519]
  We analyze the token alignments used by ROUGE and BERTScore to compare summaries. We argue that their scores largely cannot be interpreted as measuring information overlap.
  arXiv  Detail & Related papers  (2020-10-23T15:55:15Z)
• SueNes: A Weakly Supervised Approach to Evaluating Single-Document Summarization via Negative Sampling [25.299937353444854]
  We present a proof-of-concept study to a weakly supervised summary evaluation approach without the presence of reference summaries. Massive data in existing summarization datasets are transformed for training by pairing documents with corrupted reference summaries.
  arXiv  Detail & Related papers  (2020-05-13T15:40:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.