GenCompositor: Generative Video Compositing with Diffusion Transformer

• URL: http://arxiv.org/abs/2509.02460v1
• Date: Tue, 02 Sep 2025 16:10:13 GMT
• Title: GenCompositor: Generative Video Compositing with Diffusion Transformer
• Authors: Shuzhou Yang, Xiaoyu Li, Xiaodong Cun, Guangzhi Wang, Lingen Li, Ying Shan, Jian Zhang,
• Abstract summary: Traditional pipelines require intensive labor efforts and expert collaboration, resulting in lengthy production cycles and high manpower costs.<n>This new task strives to adaptively inject identity and motion information of foreground video to the target video in an interactive manner.<n>Experiments demonstrate that our method effectively realizes generative video compositing, outperforming existing possible solutions in fidelity and consistency.
• Score: 68.00271033575736
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video compositing combines live-action footage to create video production, serving as a crucial technique in video creation and film production. Traditional pipelines require intensive labor efforts and expert collaboration, resulting in lengthy production cycles and high manpower costs. To address this issue, we automate this process with generative models, called generative video compositing. This new task strives to adaptively inject identity and motion information of foreground video to the target video in an interactive manner, allowing users to customize the size, motion trajectory, and other attributes of the dynamic elements added in final video. Specifically, we designed a novel Diffusion Transformer (DiT) pipeline based on its intrinsic properties. To maintain consistency of the target video before and after editing, we revised a light-weight DiT-based background preservation branch with masked token injection. As to inherit dynamic elements from other sources, a DiT fusion block is proposed using full self-attention, along with a simple yet effective foreground augmentation for training. Besides, for fusing background and foreground videos with different layouts based on user control, we developed a novel position embedding, named Extended Rotary Position Embedding (ERoPE). Finally, we curated a dataset comprising 61K sets of videos for our new task, called VideoComp. This data includes complete dynamic elements and high-quality target videos. Experiments demonstrate that our method effectively realizes generative video compositing, outperforming existing possible solutions in fidelity and consistency.

Related papers

• Tuning-free Visual Effect Transfer across Videos [91.93897438317397]
  RefVFX is a framework that transfers complex temporal effects from a reference video onto a target video or image in a feed-forward manner.<n>We introduce a large-scale dataset of triplets, where each triplet consists of a reference effect video, an input image or video, and a corresponding output video.<n>We show that RefVFX produces visually consistent and temporally coherent edits, generalizes across unseen effect categories, and outperforms prompt-only baselines in both quantitative metrics and human preference.
  arXiv  Detail & Related papers  (2026-01-12T18:59:32Z)
• TV-LiVE: Training-Free, Text-Guided Video Editing via Layer Informed Vitality Exploitation [36.81368812919819]
  We present TV-LiVE, a Training-free and text-guided Video editing framework via Layerinformed Vitality Exploitation.<n>We empirically identify vital layers within the video generation model that significantly influence the quality of generated outputs.<n>For object addition, we identify prominent layers to extract the mask regions corresponding to the newly added target prompt.
  arXiv  Detail & Related papers  (2025-06-08T16:12:13Z)
• Resource-Efficient Motion Control for Video Generation via Dynamic Mask Guidance [2.5941932242768457]
  Mask-guided video generation can control video generation through mask motion sequences.<n>Our model enhances existing architectures by incorporating foreground masks for precise text-position matching and motion trajectory control.<n>This approach excels in various video generation tasks, such as video editing and generating artistic videos, outperforming previous methods in terms of consistency and quality.
  arXiv  Detail & Related papers  (2025-03-24T06:53:08Z)
• VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control [47.34885131252508]
  Video inpainting aims to restore corrupted video content.<n>We propose a novel dual-stream paradigm VideoPainter to process masked videos.<n>We also introduce a novel target region ID resampling technique that enables any-length video inpainting.
  arXiv  Detail & Related papers  (2025-03-07T17:59:46Z)
• Video Diffusion Transformers are In-Context Learners [31.736838809714726]
  This paper investigates a solution for enabling in-context capabilities of video diffusion transformers.<n>We propose a simple pipeline to leverage in-context generation: ($textbfii$) videos along spacial or time dimension.<n>Our framework presents a valuable tool for the research community and offers critical insights for advancing product-level controllable video generation systems.
  arXiv  Detail & Related papers  (2024-12-14T10:39:55Z)
• Video Decomposition Prior: A Methodology to Decompose Videos into Layers [74.36790196133505]
  This paper introduces a novel video decomposition prior VDP' framework which derives inspiration from professional video editing practices.<n>VDP framework decomposes a video sequence into a set of multiple RGB layers and associated opacity levels.<n>We address tasks such as video object segmentation, dehazing, and relighting.
  arXiv  Detail & Related papers  (2024-12-06T10:35:45Z)
• Anchored Diffusion for Video Face Reenactment [17.343307538702238]
  We introduce Anchored Diffusion, a novel method for synthesizing relatively long and seamless videos. We train our model on video sequences with random non-uniform temporal spacing and incorporate temporal information via external guidance. During inference, we leverage the transformer architecture to modify the diffusion process, generating a batch of non-uniform sequences anchored to a common frame.
  arXiv  Detail & Related papers  (2024-07-21T13:14:17Z)
• MagDiff: Multi-Alignment Diffusion for High-Fidelity Video Generation and Editing [90.06041718086317]
  We propose a unified Multi-alignment Diffusion, dubbed as MagDiff, for both tasks of high-fidelity video generation and editing. The proposed MagDiff introduces three types of alignments, including subject-driven alignment, adaptive prompts alignment, and high-fidelity alignment.
  arXiv  Detail & Related papers  (2023-11-29T03:36:07Z)
• SEINE: Short-to-Long Video Diffusion Model for Generative Transition and Prediction [93.26613503521664]
  This paper presents a short-to-long video diffusion model, SEINE, that focuses on generative transition and prediction. We propose a random-mask video diffusion model to automatically generate transitions based on textual descriptions. Our model generates transition videos that ensure coherence and visual quality.
  arXiv  Detail & Related papers  (2023-10-31T17:58:17Z)
• VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning [62.51232333352754]
  VideoDirectorGPT is a novel framework for consistent multi-scene video generation. Our proposed framework substantially improves layout and movement control in both single- and multi-scene video generation.
  arXiv  Detail & Related papers  (2023-09-26T17:36:26Z)
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis [73.82857768949651]
  We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled.
  arXiv  Detail & Related papers  (2021-04-30T15:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.