DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue

• URL: http://arxiv.org/abs/2504.14482v1
• Date: Sun, 20 Apr 2025 04:14:30 GMT
• Title: DialogueAgents: A Hybrid Agent-Based Speech Synthesis Framework for Multi-Party Dialogue
• Authors: Xiang Li, Duyi Pan, Hongru Xiao, Jiale Han, Jing Tang, Jiabao Ma, Wei Wang, Bo Cheng,
• Abstract summary: We propose DialogueAgents, a novel hybrid agent-based speech synthesis framework.<n>We contribute MultiTalk, a bilingual, multi-party, multi-turn speech dialogue dataset.
• Score: 17.397151329196955
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Speech synthesis is crucial for human-computer interaction, enabling natural and intuitive communication. However, existing datasets involve high construction costs due to manual annotation and suffer from limited character diversity, contextual scenarios, and emotional expressiveness. To address these issues, we propose DialogueAgents, a novel hybrid agent-based speech synthesis framework, which integrates three specialized agents -- a script writer, a speech synthesizer, and a dialogue critic -- to collaboratively generate dialogues. Grounded in a diverse character pool, the framework iteratively refines dialogue scripts and synthesizes speech based on speech review, boosting emotional expressiveness and paralinguistic features of the synthesized dialogues. Using DialogueAgent, we contribute MultiTalk, a bilingual, multi-party, multi-turn speech dialogue dataset covering diverse topics. Extensive experiments demonstrate the effectiveness of our framework and the high quality of the MultiTalk dataset. We release the dataset and code https://github.com/uirlx/DialogueAgents to facilitate future research on advanced speech synthesis models and customized data generation.

Related papers

• SpeechDialogueFactory: Generating High-Quality Speech Dialogue Data to Accelerate Your Speech-LLM Development [42.598003881584816]
  We introduce textscSpeechDialogueFactory, a production-ready framework for generating natural speech dialogues efficiently.<n>Our solution employs a comprehensive pipeline including metadata generation, dialogue scripting, paralinguistic-enriched utterance simulation, and natural speech synthesis with voice cloning.<n>We release our work as an open-source toolkit, alongside example datasets available in English and Chinese.
  arXiv  Detail & Related papers  (2025-03-31T08:52:21Z)
• OmniChat: Enhancing Spoken Dialogue Systems with Scalable Synthetic Data for Diverse Scenarios [45.78414948567598]
  We propose leveraging synthetic data to enhance the dialogue models across diverse scenarios.<n>We introduce ShareChatX, the first comprehensive, large-scale dataset for spoken dialogue that spans diverse scenarios.<n>We also explore critical aspects of training dialogue systems using synthetic data.
  arXiv  Detail & Related papers  (2025-01-02T17:58:23Z)
• DiaSynth: Synthetic Dialogue Generation Framework for Low Resource Dialogue Applications [18.378069426713]
  Existing research is constrained by general or niche datasets that lack sufficient scale for training dialogue systems.<n>We introduce Dia Synth - a synthetic dialogue generation framework capable of generating high-quality, contextually rich dialogues.<n>We perform our experiments by generating synthetic data using different LLMs and few-shot examples from DialogSum and SAMSum.
  arXiv  Detail & Related papers  (2024-09-25T07:03:31Z)
• A Framework for Synthetic Audio Conversations Generation using Large Language Models [0.0]
  Conversa Synth is a framework designed to generate synthetic conversation audio using large language models (LLMs) with multiple persona settings. The framework first creates diverse and coherent text-based dialogues across various topics, which are then converted into audio using text-to-speech (TTS) systems.
  arXiv  Detail & Related papers  (2024-09-02T05:09:46Z)
• Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation [55.043492250775294]
  We introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response. We also introduce MultiDialog, the first large-scale multimodal spoken dialogue corpus.
  arXiv  Detail & Related papers  (2024-06-12T04:48:36Z)
• Towards human-like spoken dialogue generation between AI agents from written dialogue [8.4989907582951]
  This study proposes CHATS - CHatty Agents Text-to-Speech - a discrete token-based system designed to generate spoken dialogues based on written dialogues. Our system can generate speech for both the speaker side and the listener side simultaneously, using only the transcription from the speaker side.
  arXiv  Detail & Related papers  (2023-10-02T11:03:20Z)
• Revisiting Conversation Discourse for Dialogue Disentanglement [88.3386821205896]
  We propose enhancing dialogue disentanglement by taking full advantage of the dialogue discourse characteristics. We develop a structure-aware framework to integrate the rich structural features for better modeling the conversational semantic context. Our work has great potential to facilitate broader multi-party multi-thread dialogue applications.
  arXiv  Detail & Related papers  (2023-06-06T19:17:47Z)
• PLACES: Prompting Language Models for Social Conversation Synthesis [103.94325597273316]
  We use a small set of expert-written conversations as in-context examples to synthesize a social conversation dataset using prompting. We perform several thorough evaluations of our synthetic conversations compared to human-collected conversations.
  arXiv  Detail & Related papers  (2023-02-07T05:48:16Z)
• A Benchmark for Understanding and Generating Dialogue between Characters in Stories [75.29466820496913]
  We present the first study to explore whether machines can understand and generate dialogue in stories. We propose two new tasks including Masked Dialogue Generation and Dialogue Speaker Recognition. We show the difficulty of the proposed tasks by testing existing models with automatic and manual evaluation on DialStory.
  arXiv  Detail & Related papers  (2022-09-18T10:19:04Z)
• HybriDialogue: An Information-Seeking Dialogue Dataset Grounded on Tabular and Textual Data [87.67278915655712]
  We present a new dialogue dataset, HybriDialogue, which consists of crowdsourced natural conversations grounded on both Wikipedia text and tables. The conversations are created through the decomposition of complex multihop questions into simple, realistic multiturn dialogue interactions.
  arXiv  Detail & Related papers  (2022-04-28T00:52:16Z)
• Interview: A Large-Scale Open-Source Corpus of Media Dialog [11.28504775964698]
  We introduce 'Interview': a large-scale (105K conversations) media dialog dataset collected from news interview transcripts. Compared to existing large-scale proxies for conversational data, language models trained on our dataset exhibit better zero-shot out-of-domain performance. 'Interview' contains speaker role annotations for each turn, facilitating the development of engaging, responsive dialog systems.
  arXiv  Detail & Related papers  (2020-04-07T02:44:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.