Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection

• URL: http://arxiv.org/abs/2609.14666v1
• Date: Sun, 13 Sep 2026 17:08:38 GMT
• Title: Exploring Multimodal Turn-Taking Cues in Face-to-Face Conversation using Voice Activity Projection
• Abstract summary: This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction.<n>Visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU)<n>Results show that visual information improves performance over the audio-only baseline, with being significantly more informative than other feature groups.
• Score: 11.660109871622367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Turn-taking is a fundamental component of spoken interaction, and while humans naturally rely on both verbal and non-verbal signals, dialogue systems usually depend on audio cues alone. This paper investigates whether visual features from face-to-face conversations can enhance turn-taking prediction beyond what is achievable from audio-only. We extend the Voice Activity Projection (VAP) model, a self-supervised transformer-based model for predicting future voice activity, by incorporating visual features extracted from the large-scale Meta Seamless Interaction dataset of dyadic face-to-face conversations. The visual features include gaze direction, head movement, body and hand pose, and facial action units (FAU). For incorporating the visual features, we explore concatenation, cross-attention fusion, delta features, and trainable gating mechanisms. Results show that visual information improves performance over the audio-only baseline, with FAU being significantly more informative than other feature groups. Body and gaze features nevertheless contribute complementary information, as the model combining all features performs best. Furthermore, results indicate that performance on specific tasks varies depending on whether training and test data come from improvised (acted) or naturalistic (non-acted) conversations.

Related papers

• Predicting Turn-Taking Outcomes in Multi-Party Conversation: Interpretable Modelling of Speech and Gaze Dynamics with Interpersonal Closeness [1.302669708492884]
  This study models how gaze and speech, together with perceived interpersonal closeness, signal conversational floor changes in free four-person dialogue.<n>Using the GaMMA corpus, we trained logistic regression models using interpretable, behaviourally motivated features extracted before each turn-taking event.<n>Results show that gaze features capture predictive structure, and that combining them with loudness improves performance.
  arXiv  Detail & Related papers  (2026-08-28T06:57:54Z)
• Let Me Look at You: Advanced Facial Expression Modeling for Conversational Speech Synthesis [36.79172273160707]
  Facial expressions encode subtle and rich affective cues that are crucial for empathetic speech interaction.<n>We propose FacialTalker, a facial-expression-aware CSS framework built upon a large language model backbone.<n>We construct VSDD-1K, a large-scale multimodal dialogue dataset collected through a fully automated pipeline from real-world Internet conversations.
  arXiv  Detail & Related papers  (2026-07-27T13:42:37Z)
• Conversational Human Audio-visual Talking Dialogue Generation [50.95039085506777]
  We propose CHAT, a new dyadic interactive audio-visual dialogue generation framework.<n>Chat unifies large language models and talking face models with interactive audio and facial behaviour refinement modules.<n>Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations.
  arXiv  Detail & Related papers  (2026-07-02T22:17:55Z)
• TAVID: Text-Driven Audio-Visual Interactive Dialogue Generation [72.46711449668814]
  We introduce TAVID, a unified framework that generates both interactive faces and conversational speech in a synchronized manner.<n>We evaluate our system across four dimensions: talking face realism, listening head responsiveness, dyadic interaction, and speech quality.
  arXiv  Detail & Related papers  (2025-12-23T12:04:23Z)
• Visual Cues Enhance Predictive Turn-Taking for Two-Party Human Interaction [12.424261968512385]
  Predictive turn-taking models (PTTMs) facilitate naturalistic human-robot interaction, yet most rely solely on speech.<n>We introduce MM-VAP, a multimodal PTTM which combines speech with visual cues including facial expression, head pose and gaze.<n>We find that it outperforms the state-of-the-art audio-only in videoconferencing interactions (84% vs. 79% hold/shift prediction accuracy).
  arXiv  Detail & Related papers  (2025-05-27T11:24:38Z)
• Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues [80.53407593586411]
  We focus on leveraging facial cues beyond the lip region for robust Audio-Visual Speech Enhancement (AVSE) We propose a Dual Attention Cooperative Framework, DualAVSE, to ignore speech-unrelated information, capture speech-related information with facial cues, and dynamically integrate it with the audio signal for AVSE.
  arXiv  Detail & Related papers  (2023-11-24T04:30:31Z)
• Visual-Aware Text-to-Speech [101.89332968344102]
  We present a new visual-aware text-to-speech (VA-TTS) task to synthesize speech conditioned on both textual inputs and visual feedback of the listener in face-to-face communication. We devise a baseline model to fuse phoneme linguistic information and listener visual signals for speech synthesis.
  arXiv  Detail & Related papers  (2023-06-21T05:11:39Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)
• An Overview of Deep-Learning-Based Audio-Visual Speech Enhancement and Separation [57.68765353264689]
  Speech enhancement and speech separation are two related tasks. Traditionally, these tasks have been tackled using signal processing and machine learning techniques. Deep learning has been exploited to achieve strong performance.
  arXiv  Detail & Related papers  (2020-08-21T17:24:09Z)
• Let's Face It: Probabilistic Multi-modal Interlocutor-aware Generation of Facial Gestures in Dyadic Settings [11.741529272872219]
  To enable more natural face-to-face interactions, conversational agents need to adapt their behavior to their interlocutors. Most existing gesture-generating systems do not utilize multi-modal cues from the interlocutor when synthesizing non-verbal behavior. We introduce a probabilistic method to synthesize interlocutor-aware facial gestures in dyadic conversations.
  arXiv  Detail & Related papers  (2020-06-11T14:11:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.