SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue

• URL: http://arxiv.org/abs/2603.16783v1
• Date: Tue, 17 Mar 2026 16:58:47 GMT
• Title: SpokenUS: A Spoken User Simulator for Task-Oriented Dialogue
• Abstract summary: We introduce textbfSpokenTOD, a spoken TOD dataset of 52,390 dialogues and 1,034 hours of speech augmented with four spoken user behaviors.<n>We present textbfSpokenUS, a spoken user simulator grounded in TOD with a dedicated architecture for barge-in.
• Score: 11.90483692004643
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Robust task-oriented spoken dialogue agents require exposure to the full diversity of how people interact through speech. Building spoken user simulators that address this requires large-scale spoken task-oriented dialogue (TOD) data encompassing spoken user behaviors, yet existing datasets are limited in scale and domain coverage, with no systematic pipeline for augmenting them. To address this, we introduce \textbf{SpokenTOD}, a spoken TOD dataset of 52,390 dialogues and 1,034 hours of speech augmented with four spoken user behaviors -- cross-turn slots, barge-in, disfluency, and emotional prosody -- across diverse speakers and domains. Building on SpokenTOD, we present \textbf{SpokenUS}, a spoken user simulator grounded in TOD with a dedicated architecture for barge-in. SpokenUS achieves comparable goal coverage to significantly larger models while substantially outperforming all baselines in Human MOS, disclosing slot values gradually across the dialogue as humans do rather than front-loading them. Further analysis confirms that SpokenUS's spoken behaviors pose meaningful challenges to downstream agents, making it a practical tool for training and evaluating more robust spoken dialogue systems.

Related papers

• Conversational Human Audio-visual Talking Dialogue Generation [50.95039085506777]
  We propose CHAT, a new dyadic interactive audio-visual dialogue generation framework.<n>Chat unifies large language models and talking face models with interactive audio and facial behaviour refinement modules.<n>Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations.
  arXiv  Detail & Related papers  (2026-07-02T22:17:55Z)
• REALTALK: A 21-Day Real-World Dataset for Long-Term Conversation [51.97224538045096]
  We introduce REALTALK, a 21-day corpus of authentic messaging app dialogues.<n>We compare EI attributes and persona consistency to understand the challenges posed by real-world dialogues.<n>Our findings reveal that models struggle to simulate a user solely from dialogue history, while fine-tuning on specific user chats improves persona emulation.
  arXiv  Detail & Related papers  (2025-02-18T20:29:01Z)
• Are LLMs Robust for Spoken Dialogues? [10.855403629160921]
  Large Pre-Trained Language Models have demonstrated state-of-the-art performance in different downstream tasks. Most of the publicly available datasets and benchmarks on task-oriented dialogues focus on written conversations. We have evaluated the performance of LLMs for spoken task-oriented dialogues on the DSTC11 test sets.
  arXiv  Detail & Related papers  (2024-01-04T14:36:38Z)
• SpokenWOZ: A Large-Scale Speech-Text Benchmark for Spoken Task-Oriented Dialogue Agents [70.08842857515141]
  SpokenWOZ is a large-scale speech-text dataset for spoken TOD.<n>Cross-turn slot and reasoning slot detection are new challenges for SpokenWOZ.
  arXiv  Detail & Related papers  (2023-05-22T13:47:51Z)
• PRESTO: A Multilingual Dataset for Parsing Realistic Task-Oriented Dialogs [39.58414649004708]
  PRESTO is a dataset of over 550K contextual multilingual conversations between humans and virtual assistants. It contains challenges that occur in real-world NLU tasks such as disfluencies, code-switching, and revisions. Our mT5 model based baselines demonstrate that the conversational phenomenon present in PRESTO are challenging to model.
  arXiv  Detail & Related papers  (2023-03-15T21:51:13Z)
• AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversation [54.29029787836075]
  Crowdsourced dialogue corpora are usually limited in scale and topic coverage due to the expensive cost of data curation. In this work, we leverage large language models for dialogue augmentation in the task of emotional support conversation (ESC) We construct AugESC, an augmented dataset for the ESC task, which largely extends the scale and topic coverage of the crowdsourced ESConv corpus.
  arXiv  Detail & Related papers  (2022-02-26T03:17:08Z)
• TOD-DA: Towards Boosting the Robustness of Task-oriented Dialogue Modeling on Spoken Conversations [24.245354500835465]
  We propose a novel model-agnostic data augmentation paradigm to boost the robustness of task-oriented dialogue modeling on spoken conversations. Our approach ranked first in both tasks of DSTC10 Track2, a benchmark for task-oriented dialogue modeling on spoken conversations.
  arXiv  Detail & Related papers  (2021-12-23T10:04:25Z)
• Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems [47.45333978381135]
  We introduce the Action-Based Conversations dataset with over 10K human-to-human dialogues containing 55 distinct user intents. We propose two additional dialog tasks, Action State Tracking and Cascading Dialogue Success, and establish a series of baselines. Empirical results demonstrate that while more sophisticated networks outperform simpler models, a considerable gap (50.8% absolute accuracy) still exists to reach human-level performance on ABCD.
  arXiv  Detail & Related papers  (2021-04-01T22:04:25Z)
• TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue [113.45485470103762]
  In this work, we unify nine human-human and multi-turn task-oriented dialogue datasets for language modeling. To better model dialogue behavior during pre-training, we incorporate user and system tokens into the masked language modeling.
  arXiv  Detail & Related papers  (2020-04-15T04:09:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.