DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries

• URL: http://arxiv.org/abs/2602.08149v1
• Date: Sun, 08 Feb 2026 22:46:22 GMT
• Title: DIAL-SUMMER: A Structured Evaluation Framework of Hierarchical Errors in Dialogue Summaries
• Authors: Sahana Ramnath, Nima Chitsazan, Mingyang Zhou, Chia-Hsuan Lee, Shi-Xiong Zhang, Stephen Rawls, Sambit Sahu, Sangwoo Cho, Xiang Ren, Genta Indra Winata, Akshaj Kumar Veldanda,
• Abstract summary: Prior works on dialogue summary evaluation largely ignore the complexities specific to this task.<n>We propose DIAL-SUMMER's taxonomy of errors to comprehensively evaluate dialogue summaries.<n>We conduct empirical analyses of these annotated errors, and observe interesting trends.
• Score: 34.92030460585731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dialogues are a predominant mode of communication for humans, and it is immensely helpful to have automatically generated summaries of them (e.g., to revise key points discussed in a meeting, to review conversations between customer agents and product users). Prior works on dialogue summary evaluation largely ignore the complexities specific to this task: (i) shift in structure, from multiple speakers discussing information in a scattered fashion across several turns, to a summary's sentences, and (ii) shift in narration viewpoint, from speakers' first/second-person narration, standardized third-person narration in the summary. In this work, we introduce our framework DIALSUMMER to address the above. We propose DIAL-SUMMER's taxonomy of errors to comprehensively evaluate dialogue summaries at two hierarchical levels: DIALOGUE-LEVEL that focuses on the broader speakers/turns, and WITHIN-TURN-LEVEL that focuses on the information talked about inside a turn. We then present DIAL-SUMMER's dataset composed of dialogue summaries manually annotated with our taxonomy's fine-grained errors. We conduct empirical analyses of these annotated errors, and observe interesting trends (e.g., turns occurring in middle of the dialogue are the most frequently missed in the summary, extrinsic hallucinations largely occur at the end of the summary). We also conduct experiments on LLM-Judges' capability at detecting these errors, through which we demonstrate the challenging nature of our dataset, the robustness of our taxonomy, and the need for future work in this field to enhance LLMs' performance in the same. Code and inference dataset coming soon.

Related papers

• On Mitigating Data Sparsity in Conversational Recommender Systems [69.70761335240738]
  Conversational recommender systems (CRSs) capture user preference through textual information in dialogues.<n>They suffer from data sparsity on two fronts: the dialogue space is vast and linguistically diverse, while the item space exhibits long-tail and sparse distributions.<n>Existing methods struggle with (1) generalizing to varied dialogue expressions due to underutilization of rich textual cues, and (2) learning informative item representations under severe sparsity.
  arXiv  Detail & Related papers  (2025-07-01T06:54:51Z)
• Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding tasks [0.0]
  We propose an exploration of how incorporating task-related information can enhance the summarization process. Results show that integrating models with task-related information improves summary accuracy, even with varying word error rates.
  arXiv  Detail & Related papers  (2024-09-16T08:15:35Z)
• Instructive Dialogue Summarization with Query Aggregations [41.89962538701501]
  We introduce instruction-finetuned language models to expand the capability set of dialogue summarization models. We propose a three-step approach to synthesize high-quality query-based summarization triples. By training a unified model called InstructDS on three summarization datasets with multi-purpose instructive triples, we expand the capability of dialogue summarization models.
  arXiv  Detail & Related papers  (2023-10-17T04:03:00Z)
• DIONYSUS: A Pre-trained Model for Low-Resource Dialogue Summarization [127.714919036388]
  DIONYSUS is a pre-trained encoder-decoder model for summarizing dialogues in any new domain. Our experiments show that DIONYSUS outperforms existing methods on six datasets.
  arXiv  Detail & Related papers  (2022-12-20T06:21:21Z)
• TODSum: Task-Oriented Dialogue Summarization with State Tracking [16.87549093925514]
  We introduce a large-scale public Task-Oriented Dialogue Summarization dataset, TODSum. Compared to existing work, TODSum suffers from severe scattered information issues and requires strict factual consistency. We propose a state-aware structured dialogue summarization model to integrate dialogue state information and dialogue history.
  arXiv  Detail & Related papers  (2021-10-25T06:53:11Z)
• CSDS: A Fine-grained Chinese Dataset for Customer Service Dialogue Summarization [44.21084429627218]
  We introduce a novel Chinese dataset for Customer Service Dialogue Summarization (CSDS) CSDS improves the abstractive summaries in two aspects: (1) In addition to the overall summary for the whole dialogue, role-oriented summaries are also provided to acquire different speakers' viewpoints. We compare various summarization methods on CSDS, and experiment results show that existing methods are prone to generate redundant and incoherent summaries.
  arXiv  Detail & Related papers  (2021-08-30T11:56:58Z)
• Controllable Abstractive Dialogue Summarization with Sketch Supervision [56.59357883827276]
  Our model achieves state-of-the-art performance on the largest dialogue summarization corpus SAMSum, with as high as 50.79 in ROUGE-L score.
  arXiv  Detail & Related papers  (2021-05-28T19:05:36Z)
• Unsupervised Summarization for Chat Logs with Topic-Oriented Ranking and Context-Aware Auto-Encoders [59.038157066874255]
  We propose a novel framework called RankAE to perform chat summarization without employing manually labeled data. RankAE consists of a topic-oriented ranking strategy that selects topic utterances according to centrality and diversity simultaneously. A denoising auto-encoder is designed to generate succinct but context-informative summaries based on the selected utterances.
  arXiv  Detail & Related papers  (2020-12-14T07:31:17Z)
• Multi-View Sequence-to-Sequence Models with Conversational Structure for Abstractive Dialogue Summarization [72.54873655114844]
  Text summarization is one of the most challenging and interesting problems in NLP. This work proposes a multi-view sequence-to-sequence model by first extracting conversational structures of unstructured daily chats from different views to represent conversations. Experiments on a large-scale dialogue summarization corpus demonstrated that our methods significantly outperformed previous state-of-the-art models via both automatic evaluations and human judgment.
  arXiv  Detail & Related papers  (2020-10-04T20:12:44Z)
• Unsupervised Abstractive Dialogue Summarization for Tete-a-Tetes [49.901984490961624]
  We propose the first unsupervised abstractive dialogue summarization model for tete-a-tetes (SuTaT) SuTaT consists of a conditional generative module and two unsupervised summarization modules. Experimental results show that SuTaT is superior on unsupervised dialogue summarization for both automatic and human evaluations.
  arXiv  Detail & Related papers  (2020-09-15T03:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.