Multi-human Interactive Talking Dataset

• URL: http://arxiv.org/abs/2508.03050v1
• Date: Tue, 05 Aug 2025 03:54:18 GMT
• Title: Multi-human Interactive Talking Dataset
• Authors: Zeyu Zhu, Weijia Wu, Mike Zheng Shou,
• Abstract summary: We introduce MIT, a large-scale dataset specifically designed for multi-human talking video generation.<n>The resulting dataset comprises 12 hours of high-resolution footage, each featuring two to four speakers.<n>It captures natural conversational dynamics in multi-speaker scenario, offering a rich resource for studying interactive visual behaviors.
• Score: 20.920129008402718
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing studies on talking video generation have predominantly focused on single-person monologues or isolated facial animations, limiting their applicability to realistic multi-human interactions. To bridge this gap, we introduce MIT, a large-scale dataset specifically designed for multi-human talking video generation. To this end, we develop an automatic pipeline that collects and annotates multi-person conversational videos. The resulting dataset comprises 12 hours of high-resolution footage, each featuring two to four speakers, with fine-grained annotations of body poses and speech interactions. It captures natural conversational dynamics in multi-speaker scenario, offering a rich resource for studying interactive visual behaviors. To demonstrate the potential of MIT, we furthur propose CovOG, a baseline model for this novel task. It integrates a Multi-Human Pose Encoder (MPE) to handle varying numbers of speakers by aggregating individual pose embeddings, and an Interactive Audio Driver (IAD) to modulate head dynamics based on speaker-specific audio features. Together, these components showcase the feasibility and challenges of generating realistic multi-human talking videos, establishing MIT as a valuable benchmark for future research. The code is avalibale at: https://github.com/showlab/Multi-human-Talking-Video-Dataset.

Related papers

• Seamless Interaction: Dyadic Audiovisual Motion Modeling and Large-Scale Dataset [113.25650486482762]
  We introduce the Seamless Interaction dataset, a large-scale collection of over 4,000 hours of face-to-face interaction footage.<n>This dataset enables the development of AI technologies that understand dyadic embodied dynamics.<n>We develop a suite of models that utilize the dataset to generate dyadic motion gestures and facial expressions aligned with human speech.
  arXiv  Detail & Related papers  (2025-06-27T18:09:49Z)
• Let Them Talk: Audio-Driven Multi-Person Conversational Video Generation [34.15566431966277]
  We propose a novel task: Multi-Person Conversational Video Generation.<n>We introduce a new framework, MultiTalk, to address the challenges during multi-person generation.
  arXiv  Detail & Related papers  (2025-05-28T17:57:06Z)
• MM-Conv: A Multi-modal Conversational Dataset for Virtual Humans [4.098892268127572]
  We present a novel dataset captured using a VR headset to record conversations between participants within a physics simulator (AI2-THOR) Our primary objective is to extend the field of co-speech gesture generation by incorporating rich contextual information within referential settings.
  arXiv  Detail & Related papers  (2024-09-30T21:51:30Z)
• CoVoMix: Advancing Zero-Shot Speech Generation for Human-like Multi-talker Conversations [97.75037148056367]
  CoVoMix is a novel model for zero-shot, human-like, multi-speaker, multi-round dialogue speech generation.<n>We devise a comprehensive set of metrics for measuring the effectiveness of dialogue modeling and generation.
  arXiv  Detail & Related papers  (2024-04-10T02:32:58Z)
• Audio is all in one: speech-driven gesture synthetics using WavLM pre-trained model [2.827070255699381]
  diffmotion-v2 is a speech-conditional diffusion-based generative model with WavLM pre-trained model. It can produce individual and stylized full-body co-speech gestures only using raw speech audio.
  arXiv  Detail & Related papers  (2023-08-11T08:03:28Z)
• Learning to Listen: Modeling Non-Deterministic Dyadic Facial Motion [89.01668641930206]
  We present a framework for modeling interactional communication in dyadic conversations. We autoregressively output multiple possibilities of corresponding listener motion. Our method organically captures the multimodal and non-deterministic nature of nonverbal dyadic interactions.
  arXiv  Detail & Related papers  (2022-04-18T17:58:04Z)
• DialogueNeRF: Towards Realistic Avatar Face-to-Face Conversation Video Generation [54.84137342837465]
  Face-to-face conversations account for the vast majority of daily conversations. Most existing methods focused on single-person talking head generation. We propose a novel unified framework based on neural radiance field (NeRF)
  arXiv  Detail & Related papers  (2022-03-15T14:16:49Z)
• Audio-Visual Fusion Layers for Event Type Aware Video Recognition [86.22811405685681]
  We propose a new model to address the multisensory integration problem with individual event-specific layers in a multi-task learning scheme. We show that our network is formulated with single labels, but it can output additional true multi-labels to represent the given videos.
  arXiv  Detail & Related papers  (2022-02-12T02:56:22Z)
• Learning to Predict Salient Faces: A Novel Visual-Audio Saliency Model [96.24038430433885]
  We propose a novel multi-modal video saliency model consisting of three branches: visual, audio and face. Experimental results show that the proposed method outperforms 11 state-of-the-art saliency prediction works.
  arXiv  Detail & Related papers  (2021-03-29T09:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.