MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild

• URL: http://arxiv.org/abs/2606.16731v2
• Date: Thu, 18 Jun 2026 23:18:59 GMT
• Title: MuVAP: Multimodal Multiparty Voice Activity Projection for Turn-taking Prediction in the Wild
• Abstract summary: We introduce MuVAP, a causal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks.<n>To address the complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state.<n> Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks.
• Score: 9.226832581510598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Current multiparty turn-taking models often rely on complex microphone arrays or multi-camera setups, limiting their applicability in human-robot interaction scenarios. We introduce MuVAP, a causal multimodal framework that extends Voice Activity Projection by grounding acoustic predictions in face tracks, enabling speaker-aware turn-taking predictions from a monaural audio stream and a single camera view. To address the combinatorial complexity of modeling multiple speakers, we propose Role-Relative Projection, which maps any N-speaker interaction onto a fixed current versus next floor-holder state. Because existing audiovisual datasets contain disruptive editing cuts that break causal tracking, we introduce the Audio-Visual Conversation Corpus, a 31-hour dataset of unedited, single-camera multiparty conversations. Evaluations demonstrate that MuVAP outperforms strong baselines on Shift-Hold and next-speaker prediction tasks across two- and three-speaker settings.

Related papers

• Multimodal Voice Activity Projection for Turn-Taking in Social Robots with Voice-Activity-Related Pretrained Encoders [5.677986854981345]
  Turn-taking prediction is a key requirement for social robots involved in human-human interaction.<n>This work presents a Multimodal Voice Activity Projection framework that extends the audio-only VAP.<n>The proposed approach builds on pretrained audio-visual backbones originally optimized for speech-related tasks.
  arXiv  Detail & Related papers  (2026-07-08T11:34:00Z)
• UAF: A Unified Audio Front-end LLM for Full-Duplex Speech Interaction [7.775050285048427]
  Speech assistant is driving artificial propagation toward human-like conversational systems.<n>Front-end components, such as voice activity detection (VAD) and turn-taking detection (TD) are crucial for speech assistant.<n>We propose the first unified audio frontend LLM (UAF) tailored for fullgressive speech systems.
  arXiv  Detail & Related papers  (2026-04-21T08:24:55Z)
• Speaker-Reasoner: Scaling Interaction Turns and Reasoning Patterns for Timestamped Speaker-Attributed ASR [18.90091826363701]
  Speaker-Reasoner is an end-to-end Speech LLM with agentic multi-turn temporal reasoning.<n>Speaker-Reasoner achieves consistent improvements over strong baselines on AliMeeting and AISHELL-4 datasets.
  arXiv  Detail & Related papers  (2026-04-03T14:52:20Z)
• AMUSE: Audio-Visual Benchmark and Alignment Framework for Agentic Multi-Speaker Understanding [73.05946667683259]
  Recent large language models (MLLMs) show strong perception but struggle in multi-speaker, dialogue-centric settings.<n>We introduce AMUSE, a benchmark designed around tasks that are inherently agentic.<n>We propose RAFT, a data-efficient agentic alignment framework that integrates reward optimization with intrinsic multimodal self-evaluation.
  arXiv  Detail & Related papers  (2025-12-18T07:01:47Z)
• MultiVox: A Benchmark for Evaluating Voice Assistants for Multimodal Interactions [70.93364531054273]
  We introduce MultiVox, the first benchmark to evaluate the ability of voice assistants to integrate spoken and visual cues.<n>Specifically, MultiVox includes 1000 human-annotated and recorded speech dialogues that encompass diverse paralinguistic features.<n>Our evaluation on 10 state-of-the-art models reveals that, although humans excel at these tasks, current models consistently struggle to produce contextually grounded responses.
  arXiv  Detail & Related papers  (2025-07-14T23:20:42Z)
• Unispeaker: A Unified Approach for Multimodality-driven Speaker Generation [66.49076386263509]
  This paper introduces UniSpeaker, a unified approach for multimodality-driven speaker generation.<n>We propose a unified voice aggregator based on KV-Former, applying soft contrastive loss to map diverse voice description modalities into a shared voice space.<n>UniSpeaker is evaluated across five tasks using the MVC benchmark, and the experimental results demonstrate that UniSpeaker outperforms previous modality-specific models.
  arXiv  Detail & Related papers  (2025-01-11T00:47:29Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)
• MAAS: Multi-modal Assignation for Active Speaker Detection [59.08836580733918]
  We present a novel approach to active speaker detection that directly addresses the multi-modal nature of the problem. Our experiments show that, an small graph data structure built from a single frame, allows to approximate an instantaneous audio-visual assignment problem.
  arXiv  Detail & Related papers  (2021-01-11T02:57:25Z)
• Target-Speaker Voice Activity Detection: a Novel Approach for Multi-Speaker Diarization in a Dinner Party Scenario [51.50631198081903]
  We propose a novel Target-Speaker Voice Activity Detection (TS-VAD) approach. TS-VAD directly predicts an activity of each speaker on each time frame. Experiments on the CHiME-6 unsegmented data show that TS-VAD achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2020-05-14T21:24:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.