OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains

• URL: http://arxiv.org/abs/2606.14702v2
• Date: Wed, 17 Jun 2026 03:32:13 GMT
• Title: OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains
• Abstract summary: Current automated pipelines for audio-visual Question Answering (QA) generally adopt a video-caption-QA'' paradigm.<n>We propose an automated data engine featuring two mechanisms: textbfEntity-Anchored Video Scripting' and textbfClue-Guided QA Generation'
• Score: 50.186434778589415
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current automated pipelines for audio-visual Question Answering (QA) generally adopt a ``video-caption-QA'' paradigm. However, these methods typically segment videos into short clips and generate separate descriptions for audio and visual modalities. This decoupled processing severs inherent associations between sounds and their visual sources, while independent clip processing often causes inconsistent descriptions of the same entity across segments. Furthermore, coupling long-text comprehension and QA synthesis into a single step often restricts models to localized events, yielding questions lacking long-term temporal connections and deep cross-modal reasoning. To address these issues, we propose an automated data engine featuring two mechanisms: (1) \textbf{Entity-Anchored Video Scripting} transforms videos into structured scripts, comprising summaries, main entity lists, and segment-wise audio-visual descriptions. The entity list serves as a global prior to ensure cross-segment referential consistency and reconstruct audio-visual associations. (2) \textbf{Clue-Guided QA Generation} prompts models to first mine cross-segment, multimodal clues from the script, and subsequently generate QA pairs based on these high-value clues. Leveraging this pipeline, we construct the instruction-tuning dataset \textbf{OmniVideo-100K} and a human-verified test set, \textbf{OmniVideo-Test}. Fine-tuning VITA-1.5, Qwen2.5-Omni-7B and Qwen3-Omni-30B on OmniVideo-100K yields performance gains of up to 20.59% on OmniVideo-Test, demonstrating strong generalization (up to 12.64% improvements) across established benchmarks like Daily-Omni and JointAVBench.

Related papers

• InteracVid: Building a Real Interactive Audio-Visual Response Dataset from Live-Chat Videos [7.003662856458999]
  We introduce textbfInteracVid, emphthe firstopen-source large-scale dataset that addresses this missing supervision<n>Everysample couples a preceding audio-visual context and an external stimulus with the realinteractive response that follows.<n>A ten-rater human study confirms that the extracted interactions are causal,natural, and temporally complete for both genuine and reconstructed queries.
  arXiv  Detail & Related papers  (2026-08-02T11:28:13Z)
• Empowering Long-form Omni-modal Understanding with Robust Audio Perception [48.23485436432239]
  AVDC (Audio-Visual Decoupled Captions) is a large-scale dataset designed to disentangle visual and auditory semantics.<n>AVDC captures both modality-specific nuances and complex cross-modalinteractions.<n>AVDC-QA-CoT is a Chain-of-Thought augmented question-answering dataset to foster audio-visualreasoning.
  arXiv  Detail & Related papers  (2026-07-11T13:04:24Z)
• LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV [39.241237759347825]
  LongAV- is a systematic benchmark for minute-long audio-visual generation.<n>It consists of 284 test cases spanning text-to-audio-video (T2AV), image-to---video (I2AV), and video-to-audio-video (V2AV)<n>The framework evaluates more than 20 fine-grained dimensions covering within-audio-segment quality, cross-segment consistency, global narrative coherence, semantic alignment, and audio-visual alignment.
  arXiv  Detail & Related papers  (2026-05-25T18:12:09Z)
• Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models [24.875526594002434]
  We present QTSplus, a visual token selection module for long video understanding scenarios.<n>It is integrated into Qwen2.5-VL and compresses the vision stream by up to textbf89% and reduces end-to-end latency by textbf28% on long videos.<n>Results show that QTSplus is an effective, general mechanism for scaling MLLMs to realworld long-video scenarios.
  arXiv  Detail & Related papers  (2025-11-14T22:41:27Z)
• MAGNET: A Multi-agent Framework for Finding Audio-Visual Needles by Reasoning over Multi-Video Haystacks [67.31276358668424]
  We introduce a novel task named AV-HaystacksQA, where the goal is to identify salient segments across different videos in response to a query and link them together to generate the most informative answer.<n> AVHaystacks is an audio-visual benchmark comprising 3100 annotated QA pairs designed to assess the capabilities of LMMs in multi-video retrieval and temporal grounding task.<n>We propose a model-agnostic, multi-agent framework to address this challenge, achieving up to 89% and 65% relative improvements over baseline methods on BLEU@4 and GPT evaluation scores in QA task on our proposed AVHaystack
  arXiv  Detail & Related papers  (2025-06-08T06:34:29Z)
• xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations [120.52120919834988]
  xGen-SynVideo-1 is a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens. DiT model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios.
  arXiv  Detail & Related papers  (2024-08-22T17:55:22Z)
• Diverse and Aligned Audio-to-Video Generation via Text-to-Video Model Adaptation [89.96013329530484]
  We consider the task of generating diverse and realistic videos guided by natural audio samples from a wide variety of semantic classes. We utilize an existing text-conditioned video generation model and a pre-trained audio encoder model. We validate our method extensively on three datasets demonstrating significant semantic diversity of audio-video samples.
  arXiv  Detail & Related papers  (2023-09-28T13:26:26Z)
• Structured Two-stream Attention Network for Video Question Answering [168.95603875458113]
  We propose a Structured Two-stream Attention network, namely STA, to answer a free-form or open-ended natural language question. First, we infer rich long-range temporal structures in videos using our structured segment component and encode text features. Then, our structured two-stream attention component simultaneously localizes important visual instance, reduces the influence of background video and focuses on the relevant text.
  arXiv  Detail & Related papers  (2022-06-02T12:25:52Z)
• Hierarchical Conditional Relation Networks for Multimodal Video Question Answering [67.85579756590478]
  Video QA adds at least two more layers of complexity - selecting relevant content for each channel in the context of a linguistic query. Conditional Relation Network (CRN) takes as input a set of tensorial objects translating into a new set of objects that encode relations of the inputs. CRN is then applied for Video QA in two forms, short-form where answers are reasoned solely from the visual content, and long-form where associated information, such as subtitles, is presented.
  arXiv  Detail & Related papers  (2020-10-18T02:31:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.