PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion

• URL: http://arxiv.org/abs/2606.01399v1
• Date: Sun, 31 May 2026 18:45:11 GMT
• Title: PAI-Studio: Cinematic Video Background Replacement with Camera-Aware Motion
• Abstract summary: We present PAI-Studio, a new reference-conditioned video synthesis task.<n> generating dynamic backgrounds aligned with foreground motion while preserving foreground identity.<n>We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task.
• Score: 51.8637101384136
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We present PAI-Studio, a new reference-conditioned video synthesis task that addresses a long-standing challenge in cinematic background replacement: generating dynamic backgrounds aligned with foreground motion while preserving foreground identity, matching reference scene appearance, and achieving globally consistent illumination with realistic foreground relighting. Existing open-source systems and commercial APIs cannot simultaneously ensure motion-consistent background generation, high-fidelity foreground relighting and foreground identity preservation, often resulting in static backgrounds, inconsistent boundaries, and noticeable compositing artifacts. To bridge this gap, we build upon a Diffusion Transformer video backbone and reformulate the problem as an in-context conditional generation task. Through bidirectional attention, our model jointly captures foreground dynamics and background reference information within a unified architecture. We further construct a 30K-scale dataset sourced from high-quality films and online videos to support this task. Extensive evaluations demonstrate that our method significantly outperforms existing open-source and commercial API solutions.

Related papers

• Sparkle: Realizing Lively Instruction-Guided Video Background Replacement via Decoupled Guidance [54.34473644733862]
  Background Replacement is central to creative applications such as film production and advertising.<n>Background Replacement requires entirely new, temporally consistent scenes while maintaining accurate foreground-background interactions.<n>We introduce Sparkle, a dataset of 140K video pairs spanning five common background-change themes.
  arXiv  Detail & Related papers  (2026-05-07T16:35:34Z)
• GenCompositor: Generative Video Compositing with Diffusion Transformer [68.00271033575736]
  Traditional pipelines require intensive labor efforts and expert collaboration, resulting in lengthy production cycles and high manpower costs.<n>This new task strives to adaptively inject identity and motion information of foreground video to the target video in an interactive manner.<n>Experiments demonstrate that our method effectively realizes generative video compositing, outperforming existing possible solutions in fidelity and consistency.
  arXiv  Detail & Related papers  (2025-09-02T16:10:13Z)
• Frame In-N-Out: Unbounded Controllable Image-to-Video Generation [23.231729778050873]
  We focus on a commonly used yet underexplored cinematic technique known as Frame In and Frame Out.<n>We introduce a new dataset that is curated semi-automatically, an efficient identity-preserving motion-controllable video Diffusion Transformer architecture, and a comprehensive evaluation protocol targeting this task.
  arXiv  Detail & Related papers  (2025-05-27T17:56:07Z)
• VideoPainter: Any-length Video Inpainting and Editing with Plug-and-Play Context Control [47.34885131252508]
  Video inpainting aims to restore corrupted video content.<n>We propose a novel dual-stream paradigm VideoPainter to process masked videos.<n>We also introduce a novel target region ID resampling technique that enables any-length video inpainting.
  arXiv  Detail & Related papers  (2025-03-07T17:59:46Z)
• DynIBaR: Neural Dynamic Image-Based Rendering [79.44655794967741]
  We address the problem of synthesizing novel views from a monocular video depicting a complex dynamic scene. We adopt a volumetric image-based rendering framework that synthesizes new viewpoints by aggregating features from nearby views. We demonstrate significant improvements over state-of-the-art methods on dynamic scene datasets.
  arXiv  Detail & Related papers  (2022-11-20T20:57:02Z)
• Exploring Intra- and Inter-Video Relation for Surgical Semantic Scene Segmentation [58.74791043631219]
  We propose a novel framework STswinCL that explores the complementary intra- and inter-video relations to boost segmentation performance. We extensively validate our approach on two public surgical video benchmarks, including EndoVis18 Challenge and CaDIS dataset. Experimental results demonstrate the promising performance of our method, which consistently exceeds previous state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-03-29T05:52:23Z)
• V3GAN: Decomposing Background, Foreground and Motion for Video Generation [4.791233143264228]
  We propose a method that decomposes the task of video generation into the synthesis of foreground, background and motion. Foreground and background together describe the appearance, whereas motion specifies how the foreground moves in a video over time. We propose V3GAN, a novel three-branch generative adversarial network.
  arXiv  Detail & Related papers  (2022-03-26T13:17:45Z)
• A Good Image Generator Is What You Need for High-Resolution Video Synthesis [73.82857768949651]
  We present a framework that leverages contemporary image generators to render high-resolution videos. We frame the video synthesis problem as discovering a trajectory in the latent space of a pre-trained and fixed image generator. We introduce a motion generator that discovers the desired trajectory, in which content and motion are disentangled.
  arXiv  Detail & Related papers  (2021-04-30T15:38:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.