Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals

• URL: http://arxiv.org/abs/2505.12654v2
• Date: Tue, 20 May 2025 06:59:31 GMT
• Title: Predicting Turn-Taking and Backchannel in Human-Machine Conversations Using Linguistic, Acoustic, and Visual Signals
• Abstract summary: We propose an automatic data collection pipeline that allows us to collect and annotate over 210 hours of human conversation videos.<n>We construct a Multi-Modal Face-to-Face dataset, including over 1.5M words and corresponding turn-taking and backchannel annotations from approximately 20M frames.<n>We present an end-to-end framework that predicts the probability of turn-taking and backchannel actions from multi-modal signals.
• Score: 4.550158880305567
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper addresses the gap in predicting turn-taking and backchannel actions in human-machine conversations using multi-modal signals (linguistic, acoustic, and visual). To overcome the limitation of existing datasets, we propose an automatic data collection pipeline that allows us to collect and annotate over 210 hours of human conversation videos. From this, we construct a Multi-Modal Face-to-Face (MM-F2F) human conversation dataset, including over 1.5M words and corresponding turn-taking and backchannel annotations from approximately 20M frames. Additionally, we present an end-to-end framework that predicts the probability of turn-taking and backchannel actions from multi-modal signals. The proposed model emphasizes the interrelation between modalities and supports any combination of text, audio, and video inputs, making it adaptable to a variety of realistic scenarios. Our experiments show that our approach achieves state-of-the-art performance on turn-taking and backchannel prediction tasks, achieving a 10% increase in F1-score on turn-taking and a 33% increase on backchannel prediction. Our dataset and code are publicly available online to ease of subsequent research.

Related papers

• InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos [7.003662856458999]
  We introduce textbfInteracVid, emphthe firstopen-source large-scale dataset that addresses this missing supervision<n>Everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows.<n>A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries.
  arXiv  Detail & Related papers  (2026-08-02T11:28:13Z)
• Face-to-Face: A Video Dataset for Multi-Person Interaction Modeling [53.176242285107485]
  We introduce textbfFace-to-Face with Jimmy Fallon (F2F-JF), a 70-hour, 14k-clip dataset of two-person talk-show exchanges.<n>A semi-automatic pipeline combines multi-person tracking, speech diarization, and lightweight human verification to extract temporally aligned host/guest tracks.
  arXiv  Detail & Related papers  (2026-03-16T03:50:02Z)
• Stacked Regression using Off-the-shelf, Stimulus-tuned and Fine-tuned Neural Networks for Predicting fMRI Brain Responses to Movies (Algonauts 2025 Report) [1.7266027274320124]
  We present our submission to the Algonauts 2025 Challenge.<n>The goal is to predict fMRI brain responses to movie stimuli.<n>Our approach integrates multimodal representations from large language models, video encoders, audio models, and vision-language models.
  arXiv  Detail & Related papers  (2025-10-02T15:24:16Z)
• PreFM: Online Audio-Visual Event Parsing via Predictive Future Modeling [78.61911985138795]
  We introduce Online Audio-Visual Event Parsing (On-AVEP), a novel paradigm for parsing audio, visual, and audio-visual events by sequentially analyzing incoming video streams.<n>We propose the Predictive Future Modeling framework featured by (a) predictive multimodal future modeling to infer and integrate beneficial future audio-visual cues.<n>Experiments show PreFM significantly outperforms state-of-the-art methods by a large margin with significantly fewer parameters.
  arXiv  Detail & Related papers  (2025-05-29T06:46:19Z)
• Talking Turns: Benchmarking Audio Foundation Models on Turn-Taking Dynamics [54.03209351287654]
  We propose a novel evaluation protocol that can assess spoken dialog system's turn-taking capabilities.<n>We present the first comprehensive user study that evaluates existing spoken dialogue systems on their ability to perform turn-taking events.<n>We will open source our evaluation platform to promote the development of advanced conversational AI systems.
  arXiv  Detail & Related papers  (2025-03-03T04:46:04Z)
• Nexus-O: An Omni-Perceptive And -Interactive Model for Language, Audio, And Vision [50.23246260804145]
  We introduce textbfNexus-O, an industry-level textbfomni-perceptive and -interactive model capable of efficiently processing Audio, Image, Video, and Text data.<n>We address three key research questions: First, how can models be efficiently designed and trained to achieve tri-modal alignment, understanding and reasoning capabilities across multiple modalities?<n>Second, what approaches can be implemented to evaluate tri-modal model robustness, ensuring reliable performance and applicability in real-world scenarios?<n>Third, what strategies can be employed to curate and obtain high-quality, real-life scenario
  arXiv  Detail & Related papers  (2025-02-26T17:26:36Z)
• Yeah, Un, Oh: Continuous and Real-time Backchannel Prediction with Fine-tuning of Voice Activity Projection [24.71649541757314]
  Short backchannel utterances such as "yeah" and "oh" play a crucial role in facilitating smooth and engaging dialogue.<n>This paper proposes a novel method for real-time, continuous backchannel prediction using a fine-tuned Voice Activity Projection model.
  arXiv  Detail & Related papers  (2024-10-21T11:57:56Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)
• Joint Learning of Visual-Audio Saliency Prediction and Sound Source Localization on Multi-face Videos [101.83513408195692]
  We propose a multitask learning method for visual-audio saliency prediction and sound source localization on multi-face video. The proposed method outperforms 12 state-of-the-art saliency prediction methods, and achieves competitive results in sound source localization.
  arXiv  Detail & Related papers  (2021-11-05T14:35:08Z)
• Squeeze-Excitation Convolutional Recurrent Neural Networks for Audio-Visual Scene Classification [4.191965713559235]
  This paper presents a multi-modal model for automatic scene classification. It exploits simultaneously auditory and visual information. It has been shown to provide an excellent trade-off between prediction performance and system complexity.
  arXiv  Detail & Related papers  (2021-07-28T06:10:10Z)
• Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model [96.24038430433885]
  We propose a novel multi-modal video saliency model consisting of three branches: visual, audio and face. Experimental results show that the proposed method outperforms 11 state-of-the-art saliency prediction works.
  arXiv  Detail & Related papers  (2021-03-29T09:09:39Z)
• Speech Prediction in Silent Videos using Variational Autoencoders [29.423462898526605]
  We present a model for generating speech in a silent video. The proposed model combines recurrent neural networks and variational deep generative models to learn the auditory's conditional distribution. We demonstrate the performance of our model on the GRID dataset based on standard benchmarks.
  arXiv  Detail & Related papers  (2020-11-14T17:09:03Z)
• Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers [89.00926092864368]
  We present a semantics-controlled multi-modal shuffled Transformer reasoning framework for the audio-visual scene aware dialog task. We also present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing-semantic graph representations for every frame. Our results demonstrate state-of-the-art performances on all evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-08T02:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.