Related papers: OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

URL: http://arxiv.org/abs/2604.11102v1
Date: Mon, 13 Apr 2026 07:19:27 GMT
Title: OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video
Abstract summary: This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts.<n>We present OmniScript, an 8B- parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension.
Score: 59.391671069156274
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Current multimodal large language models (MLLMs) have demonstrated remarkable capabilities in short-form video understanding, yet translating long-form cinematic videos into detailed, temporally grounded scripts remains a significant challenge. This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts encompassing character actions, dialogues, expressions, and audio cues. To facilitate this, we construct a first-of-its-kind human-annotated benchmark and propose a temporally-aware hierarchical evaluation framework. Furthermore, we present OmniScript, an 8B-parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension. OmniScript is trained via a progressive pipeline that leverages chain-of-thought supervised fine-tuning for plot and character reasoning, followed by reinforcement learning using temporally segmented rewards. Extensive experiments demonstrate that despite its parameter efficiency, OmniScript significantly outperforms larger open-source models and achieves performance comparable to state-of-the-art proprietary models, including Gemini 3-Pro, in both temporal localization and multi-field semantic accuracy.

Related papers

MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control [66.04301887685004]
Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting.<n>We propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control.<n> MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking.
arXiv Detail & Related papers (2026-06-28T16:01:04Z)
TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions [64.27159505605312]
This paper proposes Omni Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps.<n>To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions.<n>Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro.
arXiv Detail & Related papers (2026-02-09T14:21:58Z)
The Script is All You Need: An Agentic Framework for Long-Horizon Dialogue-to-Cinematic Video Generation [95.18045807704284]
We introduce an end-to-end agentic framework for dialogue-to-cinematic-video generation.<n> ScripterAgent is trained to translate coarse dialogue into a fine-grained, executable cinematic script.<n>Our framework significantly improves script faithfulness and temporal fidelity across all tested video models.
arXiv Detail & Related papers (2026-01-25T08:10:28Z)
Toward Scalable Video Narration: A Training-free Approach Using Multimodal Large Language Models [10.585096070697348]
We introduce VideoNarrator, a novel training-free pipeline designed to generate dense video captions.<n>VideoNarrator addresses challenges by leveraging a flexible pipeline where off-the-shelf MLLMs and visual-language models can function as caption generators.<n>Our experimental results demonstrate that the synergistic interaction of these components significantly enhances the quality and accuracy of video narrations.
arXiv Detail & Related papers (2025-07-22T22:16:37Z)
MLLM as Video Narrator: Mitigating Modality Imbalance in Video Moment Retrieval [53.417646562344906]
Video Moment Retrieval (VMR) aims to localize a specific temporal segment within an untrimmed long video given a natural language query. Existing methods often suffer from inadequate training annotations, i.e., the sentence typically matches with a fraction of the prominent video content in the foreground with limited wording diversity. This intrinsic modality imbalance leaves a considerable portion of visual information remaining unaligned with text. In this work, we take an MLLM as a video narrator to generate plausible textual descriptions of the video, thereby mitigating the modality imbalance and boosting the temporal localization.
arXiv Detail & Related papers (2024-06-25T18:39:43Z)
Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding [36.20990265600332]
We introduce PU-VALOR, a comprehensive audio-visual dataset comprising over 114,000 pseudo-untrimmed videos with detailed temporal annotations.<n> PU-VALOR is derived from the large-scale but coarse-annotated audio-visual dataset VALOR, through a subtle method involving event-based video clustering.<n>We develop AVicuna, a model capable of aligning audio-visual events with temporal intervals and corresponding text tokens.
arXiv Detail & Related papers (2024-03-24T19:50:49Z)
HierVL: Learning Hierarchical Video-Language Embeddings [108.77600799637172]
HierVL is a novel hierarchical video-language embedding that simultaneously accounts for both long-term and short-term associations. We introduce a hierarchical contrastive training objective that encourages text-visual alignment at both the clip level and video level. Our hierarchical scheme yields a clip representation that outperforms its single-level counterpart as well as a long-term video representation that achieves SotA.
arXiv Detail & Related papers (2023-01-05T21:53:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.