Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents

• URL: http://arxiv.org/abs/2409.15594v1
• Date: Mon, 23 Sep 2024 23:01:31 GMT
• Title: Beyond Turn-Based Interfaces: Synchronous LLMs as Full-Duplex Dialogue Agents
• Authors: Bandhav Veluri, Benjamin N Peloquin, Bokai Yu, Hongyu Gong, Shyamnath Gollakota,
• Abstract summary: We propose Synchronous LLMs for full-sync spoken dialogue modeling. We train a model that generates meaningful and natural spoken dialogue with just 2k hours of real-world spoken dialogue data. We demonstrate the model's ability to participate in full-sync dialogue by simulating interaction between two agents trained on different datasets.
• Score: 12.555910887280199
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Despite broad interest in modeling spoken dialogue agents, most approaches are inherently "half-duplex" -- restricted to turn-based interaction with responses requiring explicit prompting by the user or implicit tracking of interruption or silence events. Human dialogue, by contrast, is "full-duplex" allowing for rich synchronicity in the form of quick and dynamic turn-taking, overlapping speech, and backchanneling. Technically, the challenge of achieving full-duplex dialogue with LLMs lies in modeling synchrony as pre-trained LLMs do not have a sense of "time". To bridge this gap, we propose Synchronous LLMs for full-duplex spoken dialogue modeling. We design a novel mechanism to integrate time information into Llama3-8b so that they run synchronously with the real-world clock. We also introduce a training recipe that uses 212k hours of synthetic spoken dialogue data generated from text dialogue data to create a model that generates meaningful and natural spoken dialogue, with just 2k hours of real-world spoken dialogue data. Synchronous LLMs outperform state-of-the-art in dialogue meaningfulness while maintaining naturalness. Finally, we demonstrate the model's ability to participate in full-duplex dialogue by simulating interaction between two agents trained on different datasets, while considering Internet-scale latencies of up to 240 ms. Webpage: https://syncllm.cs.washington.edu/.

Related papers

• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [24.68804661538364]
  Full spoken dialogue systems significantly mirror human-human interactions. achieving low latency and natural interactions is a significant challenge. End-to-end full-to-end spoken dialogue systems are a promising direction for developing efficient and natural end-to-end systems. Audio samples of dialogues generated by OmniFlatten can be found at this web site.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)
• IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities [55.11130688075417]
  We introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences. We construct a multi-turn speech-to-speech dialogue dataset named method-500k which includes nearly 500k turns of speech-to-speech dialogues.
  arXiv  Detail & Related papers  (2024-10-09T05:04:31Z)
• Language Model Can Listen While Speaking [17.584201137311286]
  Listen-while-speaking language model (LSLM) is an end-to-end system equipped with both listening and speaking channels. Our results highlight LSLM's capability to achieve duplex communication with minimal impact on existing systems.
  arXiv  Detail & Related papers  (2024-08-05T16:47:22Z)
• Beyond the Turn-Based Game: Enabling Real-Time Conversations with Duplex Models [66.24055500785657]
  Traditional turn-based chat systems prevent users from verbally interacting with system while it is generating responses. To overcome these limitations, we adapt existing LLMs to listen users while generating output and provide users with instant feedback. We build a dataset consisting of alternating time slices of queries and responses as well as covering typical feedback types in instantaneous interactions.
  arXiv  Detail & Related papers  (2024-06-22T03:20:10Z)
• Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversation [55.043492250775294]
  We introduce a novel Face-to-Face spoken dialogue model. It processes audio-visual speech from user input and generates audio-visual speech as the response. We also introduce MultiDialog, the first large-scale multimodal spoken dialogue corpus.
  arXiv  Detail & Related papers  (2024-06-12T04:48:36Z)
• A Full-duplex Speech Dialogue Scheme Based On Large Language Models [23.994130020644842]
  generative generative dialogue system capable in a full manner allowing seamless interaction. System generates tokens for inquiry responses and makes autonomous decisions to respond to wait for, or operate the user.
  arXiv  Detail & Related papers  (2024-05-29T20:05:46Z)
• Evaluating Very Long-Term Conversational Memory of LLM Agents [95.84027826745609]
  We introduce a machine-human pipeline to generate high-quality, very long-term dialogues. We equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency.
  arXiv  Detail & Related papers  (2024-02-27T18:42:31Z)
• Are LLMs Robust for Spoken Dialogues? [10.855403629160921]
  Large Pre-Trained Language Models have demonstrated state-of-the-art performance in different downstream tasks. Most of the publicly available datasets and benchmarks on task-oriented dialogues focus on written conversations. We have evaluated the performance of LLMs for spoken task-oriented dialogues on the DSTC11 test sets.
  arXiv  Detail & Related papers  (2024-01-04T14:36:38Z)
• BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues [72.65163468440434]
  This report provides a preliminary evaluation of existing large language models for human-style multi-turn chatting. We prompt large language models (LLMs) to generate a full multi-turn dialogue based on the ChatSEED, utterance by utterance. We find GPT-4 can generate human-style multi-turn dialogues with impressive quality, significantly outperforms its counterparts.
  arXiv  Detail & Related papers  (2023-10-20T16:53:51Z)
• Back to the Future: Bidirectional Information Decoupling Network for Multi-turn Dialogue Modeling [80.51094098799736]
  We propose Bidirectional Information Decoupling Network (BiDeN) as a universal dialogue encoder. BiDeN explicitly incorporates both the past and future contexts and can be generalized to a wide range of dialogue-related tasks. Experimental results on datasets of different downstream tasks demonstrate the universality and effectiveness of our BiDeN.
  arXiv  Detail & Related papers  (2022-04-18T03:51:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.