MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control

• URL: http://arxiv.org/abs/2606.29473v1
• Date: Sun, 28 Jun 2026 16:01:04 GMT
• Title: MAVIN: Multi-Shot Audio-Visual Generation with Narrative Control
• Abstract summary: Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting.<n>We propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control.<n> MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking.
• Score: 66.04301887685004
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While recent generative models produce high-fidelity videos, they struggle with the complex narrative control required for coherent multi-shot audio-visual generation. Existing methods suffer from temporal misalignment, limited controllability, and incomplete scripting. In this paper, we propose MAVIN, the first framework for multi-shot audio-visual generation with customized narrative control. To resolve temporal misalignment, we propose boundary-aware attention, which leverages hierarchical captions and boundary-aware token routing to render audio-visual elements within their respective temporal boundaries. To improve the controllability for multi-subject scenarios, we propose ID-aware propagation, utilizing identity embeddings and an identity-aware mask to bind specific identities to consistent visual appearances and vocal timbres. To provide comprehensive audio-visual narratives, we present a multi-agent scripting pipeline to transform free-form user inputs into hierarchical captions. Furthermore, we construct MAVINSet, a multi-shot audio-visual dataset for robust training and evaluation. Extensive experiments demonstrate that MAVIN achieves state-of-the-art performance, opening up a new avenue for integrating generative models into professional filmmaking workflows.

Related papers

• Vorch-Omni: Multi-Task Orchestration of Sight and Sound [51.39284238118125]
  Vorch- Omni is a unified framework for audio-visual synthesis based on an arbitrary-condition-to-arbitrary-language formulation.<n>It flexibly treats video and audio signals as either conditioning inputs or generation targets.<n>It supports over 10 tasks, including text-to-video, text-to-audio-video, image- and reference-conditioned generation, temporal extension, audio-driven generation, video transformation, and audio-visual editing.
  arXiv  Detail & Related papers  (2026-08-06T09:38:30Z)
• Aura: Consistent Multi-Subject Video Generation via VLM-Grounded Semantic Alignment [20.811984469709508]
  Aura is a unified framework for high-fidelity and identity-consistent video generation.<n>We introduce AI director-level captions that provide dense and structured descriptions of video content.<n>We build a high-quality video-subject image dataset through a dedicated data construction pipeline.
  arXiv  Detail & Related papers  (2026-07-05T13:54:33Z)
• Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation [93.44732526074876]
  We present an end-to-end framework targeted at the precise binding and seamless fusion of multiple identities.<n>Within this architecture, we propose Semantic-Anchored Multimodal Ro (SA-MRo) to anchor visual and audio reference tokens, along with TTS embeddings, to their corresponding semantic descriptions.<n>Experiments demonstrate that Omni-Context achieves state-of-the-art performance in dual-modal customized generation.
  arXiv  Detail & Related papers  (2026-05-17T14:56:52Z)
• OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video [59.391671069156274]
  This paper introduces the novel video-to-script (V2S) task, aiming to generate hierarchical, scene-by-scene scripts.<n>We present OmniScript, an 8B- parameter omni-modal (audio-visual) language model tailored for long-form narrative comprehension.
  arXiv  Detail & Related papers  (2026-04-13T07:19:27Z)
• AlcheMinT: Fine-grained Temporal Control for Multi-Reference Consistent Video Generation [58.844504598618094]
  We propose AlcheMinT, a unified framework that introduces explicit timestamps conditioning for subject-driven video generation.<n>Our approach introduces a novel positional encoding mechanism that unlocks the encoding of temporal intervals, associated in our case with subject identities.<n>We incorporate subject-descriptive text tokens to strengthen binding between visual identity and video captions, mitigating ambiguity during generation.
  arXiv  Detail & Related papers  (2025-12-11T18:59:34Z)
• PresentAgent: Multimodal Agent for Presentation Video Generation [30.274831875701217]
  We present PresentAgent, a multimodal agent that transforms long-form documents into narrated presentation videos.<n>To achieve this integration, PresentAgent employs a modular pipeline that segments the input document, plans and renders slide-style visual frames.<n>Given the complexity of evaluating such multimodal outputs, we introduce PresentEval, a unified assessment framework powered by Vision-Language Models.
  arXiv  Detail & Related papers  (2025-07-05T13:24:15Z)
• Bind-Your-Avatar: Multi-Talking-Character Video Generation with Dynamic 3D-mask-based Embedding Router [72.29811385678168]
  We introduce Bind-Your-Avatar, an MM-DiT-based model specifically designed for multi-talking-character video generation in the same scene.<n>Specifically, we propose a novel framework incorporating a fine-grained Embedding Router that binds who' and speak what' together to address the audio-to-character correspondence control.
  arXiv  Detail & Related papers  (2025-06-24T17:50:16Z)
• SkyReels-Audio: Omni Audio-Conditioned Talking Portraits in Video Diffusion Transformers [25.36460340267922]
  We present SkyReels-Audio, a unified framework for synthesizing high-fidelity and temporally coherent talking portrait videos.<n>Our framework supports infinite-length generation and editing, while enabling diverse and controllable conditioning through multimodal inputs.
  arXiv  Detail & Related papers  (2025-06-01T04:27:13Z)
• OmniTalker: One-shot Real-time Text-Driven Talking Audio-Video Generation With Multimodal Style Mimicking [22.337906095079198]
  We present OmniTalker, a unified framework that jointly generates synchronized talking audio-video content from input text.<n>Our framework adopts a dual-branch diffusion transformer (DiT) architecture, with one branch dedicated to audio generation and the other to video synthesis.
  arXiv  Detail & Related papers  (2025-04-03T09:48:13Z)
• Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers [89.00926092864368]
  We present a semantics-controlled multi-modal shuffled Transformer reasoning framework for the audio-visual scene aware dialog task. We also present a novel dynamic scene graph representation learning pipeline that consists of an intra-frame reasoning layer producing-semantic graph representations for every frame. Our results demonstrate state-of-the-art performances on all evaluation metrics.
  arXiv  Detail & Related papers  (2020-07-08T02:00:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.