Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness

• URL: http://arxiv.org/abs/2608.27988v1
• Date: Fri, 28 Aug 2026 06:57:54 GMT
• Title: Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness
• Abstract summary: This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue.<n>Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event.<n>Results show that gaze features capture predictive structure, and that combining them with loudness improves performance.
• Score: 1.302669708492884
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Smooth speaker transitions are fundamental to effective conversation and rely on an interlocutor's ability to predict when to enter the conversation. This ability depends on accurately interpreting and expressing the verbal and non-verbal cues that signal when a speaker wishes to take or relinquish the floor. The process becomes even more complex in noisy, natural, multi-party settings, with multiple interlocutors available. This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue. Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event to classify floor-transfer outcomes as gaps or overlaps. Predictors included gaze features such as transition motifs and behavioural contrasts, entropy, gaze-based addressee identity, and mutual gaze, alongside speech features derived from speaker loudness, as well as perceived interpersonal closeness (IOS) between speakers. Results show that gaze features capture predictive structure, and that combining them with loudness improves performance (ROC AUC = 0.76 +- 0.04). Loudness reflected speaker control, while gaze dispersion and addressing indexed listener readiness and competitive entry. Performance remained robust across noise conditions, indicating that gaze provides a complementary, noise-resilient cue to turn-taking dynamics.

Related papers

• Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection [11.660109871622367]
  This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction.<n>Visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU)<n>Results show that visual information improves performance over the audio-only baseline, with being significantly more informative than other feature groups.
  arXiv  Detail & Related papers  (2026-09-13T17:08:38Z)
• Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis [36.79172273160707]
  Facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction.<n>We propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone.<n>We construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations.
  arXiv  Detail & Related papers  (2026-07-27T13:42:37Z)
• Acoustic and Facial Markers of Perceived Conversational Success in Spontaneous Speech [0.0]
  We analyze a large corpus of spontaneous dyadic Zoom conversations to examine how conversational dynamics relate to perceived interaction quality.<n>Results demonstrate that entrainment reliably detected in spontaneous speech and correlates with higher perceived success.
  arXiv  Detail & Related papers  (2026-03-03T19:11:38Z)
• Covo-Audio Technical Report [61.09708870154148]
  Covo-Audio, a 7B-end LALM, directly processes continuous audio inputs and generates audio outputs within a single unified architecture.<n>Covo-Audio-Chat, a dialogue-oriented variant, demonstrates semantic strong spoken conversational abilities.
  arXiv  Detail & Related papers  (2026-02-10T14:31:11Z)
• TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation [72.46711449668814]
  We introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner.<n>We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction, and speech quality.
  arXiv  Detail & Related papers  (2025-12-23T12:04:23Z)
• Lost in Phonation: Voice Quality Variation as an Evaluation Dimension for Speech Foundation Models [22.710371114925763]
  Speech foundation models (SFMs) have enabled the direct processing of spoken language from raw audio, bypassing intermediate textual representations.<n>This capability allows SFMs to be exposed to, and potentially respond to, rich paralinguistic variations embedded in the input speech signal.<n>We introduce a new parallel dataset featuring synthesized modifications to voice quality, designed to evaluate SFM responses to creaky and breathy voice.
  arXiv  Detail & Related papers  (2025-10-29T14:44:44Z)
• Chronological Thinking in Full-Duplex Spoken Dialogue Language Models [66.84843878538207]
  Chronological Thinking aims to improve response quality in full SDLMs.<n>No additional latency: once the user stops speaking, the agent halts thinking and begins speaking without further delay.<n>Results: Experiments demonstrate the effectiveness of chronological thinking through both objective metrics and human evaluations.
  arXiv  Detail & Related papers  (2025-10-02T10:28:11Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• Aligning Spoken Dialogue Models from User Interactions [55.192134724622235]
  We propose a novel preference alignment framework to improve spoken dialogue models on realtime conversations from user interactions.<n>We create a dataset of more than 150,000 preference pairs from raw multi-turn speech conversations annotated with AI feedback.<n>Our findings shed light on the importance of a well-calibrated balance among various dynamics, crucial for natural real-time speech dialogue systems.
  arXiv  Detail & Related papers  (2025-06-26T16:45:20Z)
• Dyadic Interaction Modeling for Social Behavior Generation [6.626277726145613]
  We present an effective framework for creating 3D facial motions in dyadic interactions. The heart of our framework is Dyadic Interaction Modeling (DIM), a pre-training approach. Experiments demonstrate the superiority of our framework in generating listener motions.
  arXiv  Detail & Related papers  (2024-03-14T03:21:33Z)
• Emotional Listener Portrait: Realistic Listener Motion Simulation in Conversation [50.35367785674921]
  Listener head generation centers on generating non-verbal behaviors of a listener in reference to the information delivered by a speaker. A significant challenge when generating such responses is the non-deterministic nature of fine-grained facial expressions during a conversation. We propose the Emotional Listener Portrait (ELP), which treats each fine-grained facial motion as a composition of several discrete motion-codewords. Our ELP model can not only automatically generate natural and diverse responses toward a given speaker via sampling from the learned distribution but also generate controllable responses with a predetermined attitude.
  arXiv  Detail & Related papers  (2023-09-29T18:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.