SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control

• URL: http://arxiv.org/abs/2605.27891v1
• Date: Wed, 27 May 2026 03:16:09 GMT
• Title: SmartDirector: Keyframe-Conditioned Cinematic Video Generation with Narrative Pacing Control
• Abstract summary: We propose SmartDirector, a framework that enhances the narrative capacity of video generation models.<n>SmartDirector supports flexible generation scenarios including single-shot generation, multi-shot narrative synthesis, and video extension.
• Score: 14.692728579474169
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The narrative quality of a video fundamentally determines its perceptual value. Although existing video generation methods can produce visually appealing content, they predominantly rely on sparse conditioning signals such as text prompts or first/last frames, which limits precise control over narrative structure and temporal pacing. In this paper, we propose SmartDirector, a framework that enhances the narrative capacity of video generation models through multiple keyframes. SmartDirector supports flexible generation scenarios including single-shot generation, multi-shot narrative synthesis, and video extension. The framework operates in two stages: Director-Gen generates a low-resolution video conditioned on the provided keyframes, and Director-SR refines the output by exploiting high-resolution keyframes as semantic anchors to recover fine-grained details. To enable robust multi-keyframe training, we construct a data pipeline that curates single-shot and multi-shot sequences from movies. Extensive experiments demonstrate that SmartDirector substantially outperforms existing state-of-the-art approaches. We will release the code to facilitate further research.

Related papers

• CineWeaver: Training-Free Reference-Controllable Multi-Shot Long Video Generation for Cinematic Storytelling [111.28808966759613]
  We propose a unified framework named CineWeaver to achieve reference-controllable multi-shot long-video generation without retraining.<n> Experimental results demonstrate that CineWeaver produces high-quality cinematic videos of long durations with consistent identities, stable global appearance, and clear shot transitions.
  arXiv  Detail & Related papers  (2026-07-29T06:48:11Z)
• KeyFrame-Compass: Towards Comprehensive Evaluation of Keyframe-Conditioned Video Generation [25.29345016590953]
  Video generation increasingly relies on granular synthesis-based synthesis, where creators specify a sequence of reference images to guide generation.<n>Although recent models support multi-keyframe conditioning, it remains unclear whether they can faithfully reproduce the prescribeds while maintaining overall video quality.<n>We present KeyFrame, the first comprehensive benchmark for evaluating granular synthesis-based video generation.
  arXiv  Detail & Related papers  (2026-07-15T17:46:12Z)
• ViMax: Agentic Video Generation [55.546794487716646]
  Long-form video generation requires systematic narrative planning and visual consistency.<n>Our framework employs a hierarchical narrative engine with retrieval-augmented generation for global story coherence.<n>VLM-guided agents continuously monitor and refine both narrative coherence and visual fidelity.
  arXiv  Detail & Related papers  (2026-06-02T12:55:41Z)
• A Benchmark and Multi-Agent System for Instruction-driven Cinematic Video Compilation [75.66313073019326]
  We introduce CineBench, the first benchmark for instruction-driven cinematic video compilation.<n>We present CineAgents, a multi-agent system that reformulates cinematic video compilation into design-and-compose'' paradigm.
  arXiv  Detail & Related papers  (2026-04-12T04:39:09Z)
• Tele-Omni: a Unified Multimodal Framework for Video Generation and Editing [93.8111348452324]
  Tele- Omni is a unified framework for video generation and editing that follows multimodal instructions.<n>It supports text-to-video generation, image-to-video generation, first-last-frame video generation, in-context video generation, and in-context video editing.
  arXiv  Detail & Related papers  (2026-02-10T10:01:16Z)
• Bridging Your Imagination with Audio-Video Generation via a Unified Director [54.45375287950375]
  We argue that logical reasoning and imaginative thinking are both fundamental qualities of a film director.<n>We propose UniMAGE, a unified director model that bridges user prompts with well-structured scripts.
  arXiv  Detail & Related papers  (2025-12-29T05:56:22Z)
• STAGE: Storyboard-Anchored Generation for Cinematic Multi-shot Narrative [55.05324155854762]
  We introduce a SToryboard-Anchored GEneration workflow to reformulate the STAGE-based video generation task.<n>Instead of using sparses, we propose STEP2 to predict a structural storyboard composed of start-end frame pairs for each shot.<n>We also contribute the large-scale ConStoryBoard dataset, including high-quality movie clips with fine-grained narratives for story progression, cinematic attributes, and human preferences.
  arXiv  Detail & Related papers  (2025-12-13T15:57:29Z)
• FilmWeaver: Weaving Consistent Multi-Shot Videos with Cache-Guided Autoregressive Diffusion [46.67733869872552]
  textbfFilmWeaver is a framework designed to generate consistent, multi-shot videos of arbitrary length.<n>Our key insight is to decouple the problem into inter-shot consistency and intra-shot coherence.<n>Our method surpasses existing approaches on metrics for both consistency and aesthetic quality.
  arXiv  Detail & Related papers  (2025-12-12T04:34:53Z)
• MultiShotMaster: A Controllable Multi-Shot Video Generation Framework [67.38203939500157]
  Current generation techniques excel at single-shot clips but struggle to produce narrative multi-shot videos.<n>We propose MultiShotMaster, a framework for highly controllable multi-shot video generation.
  arXiv  Detail & Related papers  (2025-12-02T18:59:48Z)
• EchoShot: Multi-Shot Portrait Video Generation [37.77879735014084]
  EchoShot is a native multi-shot framework for portrait customization built upon a foundation video diffusion model.<n>To facilitate model training within multi-shot scenario, we construct PortraitGala, a large-scale and high-fidelity human-centric video dataset.<n>To further enhance applicability, we extend EchoShot to perform reference image-based personalized multi-shot generation and long video synthesis with infinite shot counts.
  arXiv  Detail & Related papers  (2025-06-16T11:00:16Z)
• VideoGen-of-Thought: Step-by-step generating multi-shot video with minimal manual intervention [76.3175166538482]
  VideoGen-of-Thought (VGoT) is a step-by-step framework that automates multi-shot video synthesis from a single sentence.<n>VGoT addresses three core challenges: Narrative fragmentation, visual inconsistency, and transition artifacts.<n>Combined in a training-free pipeline, VGoT surpasses strong baselines by 20.4% in within-shot face consistency and 17.4% in style consistency.
  arXiv  Detail & Related papers  (2024-12-03T08:33:50Z)
• DirecT2V: Large Language Models are Frame-Level Directors for Zero-Shot Text-to-Video Generation [37.25815760042241]
  This paper introduces a new framework, dubbed DirecT2V, to generate text-to-video (T2V) videos. We equip a diffusion model with a novel value mapping method and dual-softmax filtering, which do not require any additional training. The experimental results validate the effectiveness of our framework in producing visually coherent and storyful videos.
  arXiv  Detail & Related papers  (2023-05-23T17:57:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.