Autoregressive Modeling of Film with Applications in Video Montage

• URL: http://arxiv.org/abs/2607.14645v2
• Date: Mon, 20 Jul 2026 15:29:48 GMT
• Title: Autoregressive Modeling of Film with Applications in Video Montage
• Abstract summary: FilmGPT is an autoregressive transformer designed to turn raw, "unwatchable" footage into coherent cinematic sequences.<n>Inspired by language learning in modern LLMs, we train a long-context autoregressive transformer on a large corpus of movies.<n>We evaluate our footage-constrained decoding algorithm on the full film editing task via a user study, and find that our FilmGPT-based editing significantly outperforms previous approaches.
• Score: 55.93893329580228
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This work introduces FilmGPT, an autoregressive transformer designed to address the challenge of video montage--turning a collection of raw, "unwatchable" footage into coherent cinematic sequences. Inspired by language learning in modern LLMs, we train a long-context autoregressive transformer on a large corpus of movies. The aim is to implicitly capture the "grammar" of film directly from data rather than from hand-coded rules. Unlike other generative models, FilmGPT does not generate any new video frames. Instead, at inference time, we introduce a footage-constrained decoding algorithm to select the best next shot from the input raw footage according to the statistical patterns learned from films. We first evaluate these learned statistics directly by using the FilmGPT autoregressive model for next shot prediction on a standard benchmark of shot sequence ordering, outperforming the previous state of the art. We then evaluate our footage-constrained decoding algorithm on the full film editing task via a user study, and find that our FilmGPT-based editing significantly outperforms previous approaches. Finally, we demonstrate the applicability of FilmGPT to a wide range of applications in video montage, from automatic video segment trimming to human-in-the-loop film editing.

Related papers

• ShotDirector: Directorially Controllable Multi-Shot Video Generation with Cinematographic Transitions [46.3918771233715]
  ShotDirector is an efficient framework that integrates parameter-level camera control and hierarchical editing-pattern-aware prompting.<n>Our framework effectively combines parameter-level conditions with high-level semantic guidance, achieving film-like controllable shot transitions.
  arXiv  Detail & Related papers  (2025-12-11T05:05:07Z)
• ESA: Energy-Based Shot Assembly Optimization for Automatic Video Editing [12.967240894970098]
  Shot assembly is a crucial step in film production and video editing.<n>Traditionally, this process has been manually executed by experienced editors.<n>We propose an energy-based optimization method for video shot assembly.
  arXiv  Detail & Related papers  (2025-11-04T11:48:22Z)
• CineTrans: Learning to Generate Videos with Cinematic Transitions via Masked Diffusion Models [28.224969852134606]
  We introduce CineTrans, a framework for generating coherent multi-shot videos with cinematic, film-style transitions.<n>CineTrans produces cinematic multi-shot sequences while adhering to the film editing style, avoiding unstable transitions or naive concatenations.
  arXiv  Detail & Related papers  (2025-08-15T13:58:22Z)
• Cut2Next: Generating Next Shot via In-Context Tuning [93.14744132897428]
  Multi-shot generation demands purposeful, film-like transitions and strict cinematic continuity.<n>Current methods often prioritize basic visual consistency, neglecting crucial editing patterns.<n>We introduce Next Shot Generation (NSG): a subsequent, high-quality shot that critically synthesizes professional editing patterns.
  arXiv  Detail & Related papers  (2025-08-11T17:56:59Z)
• EditIQ: Automated Cinematic Editing of Static Wide-Angle Videos via Dialogue Interpretation and Saliency Cues [6.844857856353673]
  We present EditIQ, a completely automated framework for cinematically editing scenes captured via a stationary, large field-of-view and high-resolution camera.<n>From the static camera feed, EditIQ initially generates multiple virtual feeds, emulating a team of cameramen.<n>These virtual camera shots termed rushes are subsequently assembled using an automated editing algorithm, whose objective is to present the viewer with the most vivid scene content.
  arXiv  Detail & Related papers  (2025-02-04T09:45:52Z)
• Video Decomposition Prior: A Methodology to Decompose Videos into Layers [74.36790196133505]
  This paper introduces a novel video decomposition prior VDP' framework which derives inspiration from professional video editing practices.<n>VDP framework decomposes a video sequence into a set of multiple RGB layers and associated opacity levels.<n>We address tasks such as video object segmentation, dehazing, and relighting.
  arXiv  Detail & Related papers  (2024-12-06T10:35:45Z)
• SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction [93.26613503521664]
  This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. We propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. Our model generates transition videos that ensure coherence and visual quality.
  arXiv  Detail & Related papers  (2023-10-31T17:58:17Z)
• Ground-A-Video: Zero-shot Grounded Video Editing using Text-to-image Diffusion Models [65.268245109828]
  Ground-A-Video is a video-to-video translation framework for multi-attribute video editing. It attains temporally consistent editing of input videos in a training-free manner. Experiments and applications demonstrate that Ground-A-Video's zero-shot capacity outperforms other baseline methods in terms of edit-accuracy and frame consistency.
  arXiv  Detail & Related papers  (2023-10-02T11:28:37Z)
• Dreamix: Video Diffusion Models are General Video Editors [22.127604561922897]
  Text-driven image and video diffusion models have recently achieved unprecedented generation realism. We present the first diffusion-based method that is able to perform text-based motion and appearance editing of general videos.
  arXiv  Detail & Related papers  (2023-02-02T18:58:58Z)
• The Anatomy of Video Editing: A Dataset and Benchmark Suite for AI-Assisted Video Editing [90.59584961661345]
  This work introduces the Anatomy of Video Editing, a dataset, and benchmark, to foster research in AI-assisted video editing. Our benchmark suite focuses on video editing tasks, beyond visual effects, such as automatic footage organization and assisted video assembling. To enable research on these fronts, we annotate more than 1.5M tags, with relevant concepts to cinematography, from 196176 shots sampled from movie scenes.
  arXiv  Detail & Related papers  (2022-07-20T10:53:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.