Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization

• URL: http://arxiv.org/abs/2512.14687v2
• Date: Wed, 17 Dec 2025 23:02:10 GMT
• Title: Spoken DialogSum: An Emotion-Rich Conversational Dataset for Spoken Dialogue Summarization
• Authors: Yen-Ju Lu, Kunxiao Gao, Mingrui Liang, Helin Wang, Thomas Thebaud, Laureano Moro-Velazquez, Najim Dehak, Jesus Villalba,
• Abstract summary: Spoken DialogSum is the first corpus aligning raw conversational audio with factual summaries, emotion-rich summaries, and utterance-level labels.<n>The dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with emotion, pitch, and speaking rate.<n> Spoken DialogSum comprises 13,460 emotion-diverse dialogues, each paired with both a factual and an emotion-focused summary.
• Score: 21.32336226752075
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent audio language models can follow long conversations. However, research on emotion-aware or spoken dialogue summarization is constrained by the lack of data that links speech, summaries, and paralinguistic cues. We introduce Spoken DialogSum, the first corpus aligning raw conversational audio with factual summaries, emotion-rich summaries, and utterance-level labels for speaker age, gender, and emotion. The dataset is built in two stages: first, an LLM rewrites DialogSum scripts with Switchboard-style fillers and back-channels, then tags each utterance with emotion, pitch, and speaking rate. Second, an expressive TTS engine synthesizes speech from the tagged scripts, aligned with paralinguistic labels. Spoken DialogSum comprises 13,460 emotion-diverse dialogues, each paired with both a factual and an emotion-focused summary. We release an online demo at https://fatfat-emosum.github.io/EmoDialog-Sum-Audio-Samples/, with plans to release the full dataset in the near future. Baselines show that an Audio-LLM raises emotional-summary ROUGE-L by 28% relative to a cascaded ASR-LLM system, confirming the value of end-to-end speech modeling.

Related papers

• ZipVoice-Dialog: Non-Autoregressive Spoken Dialogue Generation with Flow Matching [22.477986192421767]
  We introduce ZipVoice-Dialog, a non-autoregressive spoken dialogue generation model built upon flow matching.<n>Key designs include speaker-turn embeddings for precise speaker turn-taking.<n>We curated OpenDialog, a 6.8k-hour spoken dialogue dataset from in-the-wild speech data.
  arXiv  Detail & Related papers  (2025-07-12T15:18:47Z)
• DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue [17.397151329196955]
  We propose DialogueAgents, a novel hybrid agent-based speech synthesis framework.<n>We contribute MultiTalk, a bilingual, multi-party, multi-turn speech dialogue dataset.
  arXiv  Detail & Related papers  (2025-04-20T04:14:30Z)
• Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis [39.25088200618052]
  Conversational speech synthesis (CSS) aims to take the current dialogue (CD) history as a reference to synthesize expressive speech that aligns with the conversational style.<n>Unlike CD, stored dialogue (SD) contains preserved dialogue fragments from earlier stages of user-agent interaction.<n>This knowledge plays a significant role in enabling the agent to synthesize expressive conversational speech that generates empathetic feedback.
  arXiv  Detail & Related papers  (2025-01-11T07:43:18Z)
• OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios [45.78414948567598]
  We propose leveraging synthetic data to enhance the dialogue models across diverse scenarios.<n>We introduce ShareChatX, the first comprehensive, large-scale dataset for spoken dialogue that spans diverse scenarios.<n>We also explore critical aspects of training dialogue systems using synthetic data.
  arXiv  Detail & Related papers  (2025-01-02T17:58:23Z)
• SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation [56.683846056788326]
  We propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration.<n>We convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme.<n> Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence.
  arXiv  Detail & Related papers  (2025-01-01T11:11:07Z)
• WavChat: A Survey of Spoken Dialogue Models [66.82775211793547]
  Recent advancements in spoken dialogue models, exemplified by systems like GPT-4o, have captured significant attention in the speech domain. These advanced spoken dialogue models not only comprehend audio, music, and other speech-related features, but also capture stylistic and timbral characteristics in speech. Despite the progress in spoken dialogue systems, there is a lack of comprehensive surveys that systematically organize and analyze these systems.
  arXiv  Detail & Related papers  (2024-11-15T04:16:45Z)
• Moshi: a speech-text foundation model for real-time dialogue [78.88479749811376]
  Current systems for spoken dialogue rely on pipelines independent voice activity detection and text-to-speech. We show how Moshi Moshi can provide streaming speech recognition and text-to-speech. Our resulting model is first real-time full spoken large language model modality.
  arXiv  Detail & Related papers  (2024-09-17T17:55:39Z)
• Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation [55.043492250775294]
  We introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response. We also introduce MultiDialog, the first large-scale multimodal spoken dialogue corpus.
  arXiv  Detail & Related papers  (2024-06-12T04:48:36Z)
• Chat-Capsule: A Hierarchical Capsule for Dialog-level Emotion Analysis [70.98130990040228]
  We propose a Context-based Hierarchical Attention Capsule(Chat-Capsule) model, which models both utterance-level and dialog-level emotions and their interrelations. On a dialog dataset collected from customer support of an e-commerce platform, our model is also able to predict user satisfaction and emotion curve category.
  arXiv  Detail & Related papers  (2022-03-23T08:04:30Z)
• EMOVIE: A Mandarin Emotion Speech Dataset with a Simple Emotional Text-to-Speech Model [56.75775793011719]
  We introduce and publicly release a Mandarin emotion speech dataset including 9,724 samples with audio files and its emotion human-labeled annotation. Unlike those models which need additional reference audio as input, our model could predict emotion labels just from the input text and generate more expressive speech conditioned on the emotion embedding. In the experiment phase, we first validate the effectiveness of our dataset by an emotion classification task. Then we train our model on the proposed dataset and conduct a series of subjective evaluations.
  arXiv  Detail & Related papers  (2021-06-17T08:34:21Z)
• Interview: A Large-Scale Open-Source Corpus of Media Dialog [11.28504775964698]
  We introduce 'Interview': a large-scale (105K conversations) media dialog dataset collected from news interview transcripts. Compared to existing large-scale proxies for conversational data, language models trained on our dataset exhibit better zero-shot out-of-domain performance. 'Interview' contains speaker role annotations for each turn, facilitating the development of engaging, responsive dialog systems.
  arXiv  Detail & Related papers  (2020-04-07T02:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.