InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos

• URL: http://arxiv.org/abs/2608.01157v1
• Date: Sun, 02 Aug 2026 11:28:13 GMT
• Title: InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos
• Abstract summary: We introduce textbfInteracVid, emphthe firstopen-source large-scale dataset that addresses this missing supervision<n>Everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows.<n>A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries.
• Score: 7.003662856458999
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models have made text the default medium for human--AI interaction, buttext alone cannot express the full range of responses required by multimodal assistants,avatars, and embodied agents. While recent audio-video generative models can synthesizehigh-fidelity synchronized content, existing supervision is largely \emph{descriptive}:models are trained to render captions rather than to produce audio-visual responsescaused by external user interactions. We introduce \textbf{InteracVid}, \emph{the firstopen-source large-scale dataset that addresses this missing supervision}, so that everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows. We design a metadata-aware pipeline that extractsinteractive clips from long, noisy livestreams, yielding over \textbf{454K}context-query-response triplets from more than \textbf{59K} livestream videos andspanning conversation-centered, object-centric, procedural, embodied, and screen-basedscenarios. A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries. On aheld-out benchmark of \textbf{100} genuine live-chat queries, fine-tuning on InteracVidimproves both interaction planning and audio-video response generation, and anindependent human evaluation reproduces the system ranking and the conclusions obtainedwith our automatic judge. These results highlight interaction-structured data as acritical foundation for interactive multimodal generation.

Related papers

• Conversational Human Audio-visual Talking Dialogue Generation [50.95039085506777]
  We propose CHAT, a new dyadic interactive audio-visual dialogue generation framework.<n>Chat unifies large language models and talking face models with interactive audio and facial behaviour refinement modules.<n>Experiments show that CHAT outperforms existing related methods designed for similar tasks under both objective and subjective evaluations.
  arXiv  Detail & Related papers  (2026-07-02T22:17:55Z)
• OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains [50.186434778589415]
  Current automated pipelines for audio-visual Question Answering (QA) generally adopt a video-caption-QA'' paradigm.<n>We propose an automated data engine featuring two mechanisms: textbfEntity-Anchored Video Scripting' and textbfClue-Guided QA Generation'
  arXiv  Detail & Related papers  (2026-06-12T17:59:55Z)
• ATIR: Towards Audio-Text Interleaved Contextual Retrieval [63.68521448682396]
  We introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities.<n>This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval.
  arXiv  Detail & Related papers  (2026-04-22T07:11:58Z)
• InterDyad: Interactive Dyadic Speech-to-Video Generation by Querying Intermediate Visual Guidance [20.740979380270126]
  We propose InterDyad, a synthesis framework for interactive dyadic dynamics.<n>We first design an Interactivity that achieves video reenactment based on identity-agnostic motion priors extracted from reference videos.<n>By leveraging a Multimodal Large Language Model (MLLM), our framework is able to distill linguistic intent from audio to dictate the precise timing and appropriateness of reactions.<n> Comprehensive experiments demonstrate that InterDyad significantly outperforms state-of-the-art methods in producing natural and contextually grounded two-person interactions.
  arXiv  Detail & Related papers  (2026-03-24T12:27:52Z)
• Face-to-Face: A Video Dataset for Multi-Person Interaction Modeling [53.176242285107485]
  We introduce textbfFace-to-Face with Jimmy Fallon (F2F-JF), a 70-hour, 14k-clip dataset of two-person talk-show exchanges.<n>A semi-automatic pipeline combines multi-person tracking, speech diarization, and lightweight human verification to extract temporally aligned host/guest tracks.
  arXiv  Detail & Related papers  (2026-03-16T03:50:02Z)
• ChatUMM: Robust Context Tracking for Conversational Interleaved Generation [44.19929499646892]
  Unified multimodal models (UMMs) have achieved remarkable progress yet remain constrained by a single-turn interaction paradigm.<n>We present ChatUMM, a conversational unified model that excels at robust context tracking to sustain interleaved multimodal generation.<n>ChatUMM derives its capabilities from an interleaved multi-turn training strategy that models serialized text-image streams as a continuous conversational flow.
  arXiv  Detail & Related papers  (2026-02-06T07:11:50Z)
• OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions [62.19092662469285]
  Online Multimodal Conversational Response Generation (OMCRG) is a novel task designed to produce synchronized verbal and non-verbal listener feedback online.<n>We propose OmniResponse, a Multimodal Large Language Model (MLLM) that autoregressively generates accurate multimodal listener responses.<n>We offer ResponseNet, a dataset of 696 detailed dyadic interactions featuring synchronized split-screen videos, multichannel audio, transcripts, and annotated facial behaviors.
  arXiv  Detail & Related papers  (2025-05-27T20:12:46Z)
• Multi-party Response Generation with Relation Disentanglement [8.478506896774137]
  Existing neural response generation models have achieved impressive improvements for two-party conversations. However, many real-world dialogues involve multiple interlocutors and the structure of conversational context is much more complex. We propose to automatically infer the relations via relational thinking on subtle clues inside the conversation context without any human label.
  arXiv  Detail & Related papers  (2024-03-16T06:33:44Z)
• Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers [89.00926092864368]
  We present a semantics-controlled multi-modal shuffled Transformer reasoning framework for the audio-visual scene aware dialog task. We also present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing-semantic graph representations for every frame. Our results demonstrate state-of-the-art performances on all evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-08T02:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.