DiscussLLM: Teaching Large Language Models When to Speak

• URL: http://arxiv.org/abs/2508.18167v1
• Date: Mon, 25 Aug 2025 16:16:42 GMT
• Title: DiscussLLM: Teaching Large Language Models When to Speak
• Authors: Deep Anil Patel, Iain Melvin, Christopher Malon, Martin Renqiang Min,
• Abstract summary: Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like text.<n>We introduce $textitDiscussLLM$, a framework designed to bridge this gap by training models to proactively decide not just $textitwhat$ to say, but critically, $textitwhen$ to speak.
• Score: 9.441455921296301
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Large Language Models (LLMs) have demonstrated remarkable capabilities in understanding and generating human-like text, yet they largely operate as reactive agents, responding only when directly prompted. This passivity creates an "awareness gap," limiting their potential as truly collaborative partners in dynamic human discussions. We introduce $\textit{DiscussLLM}$, a framework designed to bridge this gap by training models to proactively decide not just $\textit{what}$ to say, but critically, $\textit{when}$ to speak. Our primary contribution is a scalable two-stage data generation pipeline that synthesizes a large-scale dataset of realistic multi-turn human discussions. Each discussion is annotated with one of five intervention types (e.g., Factual Correction, Concept Definition) and contains an explicit conversational trigger where an AI intervention adds value. By training models to predict a special silent token when no intervention is needed, they learn to remain quiet until a helpful contribution can be made. We explore two architectural baselines: an integrated end-to-end model and a decoupled classifier-generator system optimized for low-latency inference. We evaluate these models on their ability to accurately time interventions and generate helpful responses, paving the way for more situationally aware and proactive conversational AI.

Related papers

• F-Actor: Controllable Conversational Behaviour in Full-Duplex Models [70.48189107402145]
  We present first open, instruction-following full-stage conversational speech model that can be trained efficiently under typical academic resource constraints.<n>Our model requires just 2,000 hours of data, without relying on large-scale pretraining or multi-stage pretraining.<n>Both the model and training code will be released to enable reproducible research on controllable full-like controllable full-stage speech systems.
  arXiv  Detail & Related papers  (2026-01-16T14:25:57Z)
• FLEXI: Benchmarking Full-duplex Human-LLM Speech Interaction [49.83226596963294]
  Speech-computer human interaction enables real-time spoken dialogue systems.<n>Modelling and benchmarking these models remains a fundamental challenge.<n>We introduce FLEXI, the first benchmark for full-human spoken interaction.
  arXiv  Detail & Related papers  (2025-09-26T11:57:42Z)
• UniConv: Unifying Retrieval and Response Generation for Large Language Models in Conversations [71.79210031338464]
  We show how to unify dense retrieval and response generation for large language models in conversation.<n>We conduct joint fine-tuning with different objectives and design two mechanisms to reduce the inconsistency risks.<n>The evaluations on five conversational search datasets demonstrate that our unified model can mutually improve both tasks and outperform the existing baselines.
  arXiv  Detail & Related papers  (2025-07-09T17:02:40Z)
• Aligning Spoken Dialogue Models from User Interactions [55.192134724622235]
  We propose a novel preference alignment framework to improve spoken dialogue models on realtime conversations from user interactions.<n>We create a dataset of more than 150,000 preference pairs from raw multi-turn speech conversations annotated with AI feedback.<n>Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.
  arXiv  Detail & Related papers  (2025-06-26T16:45:20Z)
• Towards Anthropomorphic Conversational AI Part I: A Practical Framework [49.62013440962072]
  We introduce a multi- module framework designed to replicate the key aspects of human intelligence involved in conversations.<n>In the second stage of our approach, these conversational data, after filtering and labeling, can serve as training and testing data for reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-28T03:18:39Z)
• VAGUE: Visual Contexts Clarify Ambiguous Expressions [15.140825578254324]
  We introduce VAGUE, a benchmark evaluating multimodal AI systems' ability to integrate visual context for intent.<n>VAGUE consists of 1.6K ambiguous textual expressions, each paired with an image and multiple-choice interpretations.<n>Our experiments reveal that existing multimodal AI models struggle to infer the speaker's true intent.
  arXiv  Detail & Related papers  (2024-11-21T14:01:42Z)
• OmniFlatten: An End-to-end GPT Model for Seamless Voice Conversation [53.7173034249361]
  End-to-end GPT-based model OmniFlatten capable of effectively modeling complex behaviors inherent natural conversations with low latency.<n>Our approach offers a simple modeling technique and a promising research direction for developing efficient and natural end-to-end full- spoken dialogue systems.
  arXiv  Detail & Related papers  (2024-10-23T11:58:58Z)
• Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming [0.0]
  Mini- Omni is an audio-based end-to-end conversational model capable of real-time speech interaction. We propose a text-instructed speech generation method, along with batch-parallel strategies during inference to boost the performance. We also introduce the VoiceAssistant-400K dataset to fine-tune models for optimized speech output.
  arXiv  Detail & Related papers  (2024-08-29T17:18:53Z)
• Pre-training Multi-party Dialogue Models with Latent Discourse Inference [85.9683181507206]
  We pre-train a model that understands the discourse structure of multi-party dialogues, namely, to whom each utterance is replying. To fully utilize the unlabeled data, we propose to treat the discourse structures as latent variables, then jointly infer them and pre-train the discourse-aware model.
  arXiv  Detail & Related papers  (2023-05-24T14:06:27Z)
• Gated Multimodal Fusion with Contrastive Learning for Turn-taking Prediction in Human-robot Dialogue [15.710861456924158]
  Turn-taking, aiming to decide when the next speaker can start talking, is an essential component in building human-robot spoken dialogue systems. We first collect a large-scale annotated corpus for turn-taking with over 5,000 real human-robot dialogues in speech and text modalities. A novel gated multimodal fusion mechanism is devised to utilize various information seamlessly for turn-taking prediction.
  arXiv  Detail & Related papers  (2022-04-18T05:18:00Z)
• TANet: Thread-Aware Pretraining for Abstractive Conversational Summarization [27.185068253347257]
  We build a large-scale (11M) pretraining dataset called RCS based on the multi-person discussions in the Reddit community. We then present TANet, a thread-aware Transformer-based network. Unlike the existing pre-trained models that treat a conversation as a sequence of sentences, we argue that the inherent contextual dependency plays an essential role in understanding the entire conversation.
  arXiv  Detail & Related papers  (2022-04-09T16:08:46Z)
• Plug-and-Play Conversational Models [62.77150879036442]
  We introduce an approach that does not require further computation at decoding time, while also does not require any fine-tuning of a large language model. We demonstrate, through extensive automatic and human evaluation, a high degree of control over the generated conversational responses with regard to multiple desired attributes.
  arXiv  Detail & Related papers  (2020-10-09T03:17:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.