Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models

• URL: http://arxiv.org/abs/2606.20233v1
• Date: Wed, 17 Jun 2026 17:56:01 GMT
• Title: Cinematic Compositing Using Character-Environment-Harmonized Video Generation Models
• Abstract summary: Cinematic compositing aims to integrate green-screen characters into novel environments while maintaining physical and photometric realism.<n>Previous methods often fail to capture the complex bidirectional interactions between characters and their surroundings.<n>We propose an end-to-end video diffusion framework that jointly models C2E and E2C interactions.
• Score: 18.132403462982335
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Cinematic compositing aims to integrate green-screen characters into novel environments while maintaining physical and photometric realism. Previous methods often fail to capture the complex bidirectional interactions between characters and their surroundings, which we characterize as Character-to-Environment (C2E) physical interaction and Environment-to-Character (E2C) lighting harmonization. To address this, we propose an end-to-end video diffusion framework that jointly models C2E and E2C interactions, specifically handling the challenges of interactive props. Our approach introduces a tri-mask-guided architecture with RGB-D joint denoising to ensure physically consistent interactions among the character, props, and environment. We further develop an efficient prior-driven data curation pipeline to construct high-quality relighting pairs without expensive rendering. Finally, a reference-conditioned mechanism enables controllable environment synthesis and precise prop replacement. Extensive experiments demonstrate that our framework significantly outperforms existing methods in cinematic-quality dynamic video compositing.

Related papers

• Dynamic-Robust Photometric-Semantic Reconstruction for Open-Vocabulary 3D Scene Understanding [63.599342750566485]
  SPAR is a novel joint semantic-geometric encoding architecture that explicitly isolates transient dynamic noise prior to latent space aggregation.<n>Our end-to-end approach achieves exceptional novel view synthesis quality, yielding a PSNR of 22.15 dB and 23.33 dB given only 3 and 4 input views respectively.
  arXiv  Detail & Related papers  (2026-08-29T09:58:24Z)
• FlexComposer: Unified Video Compositing from Images to Dynamic Footage with Flexible Trajectory Control [26.208912960696622]
  We propose a unified framework that standardizes video compositing as a trajectory-guided conditional generation task.<n>Our approach introduces three key designs: (1) a Unified Foreground Representation that decouples an object's intrinsic motion from its global displacement, standardizing heterogeneous inputs into a stabilized, centered latent space; (2) a Spatial-Aware Latent Injection strategy that exploits the translation equivariance of VAE latent spaces to transport canonical features onto target trajectories via a parameter-free mechanism; and (3) a Hybrid dataset and Synthetic-to-Real Curriculum that synergizes procedural simulation, real-world cinematic footage, and generative data to
  arXiv  Detail & Related papers  (2026-07-31T16:59:46Z)
• CoInteract: Physically-Consistent Human-Object Interaction Video Synthesis via Spatially-Structured Co-Generation [23.278316670516897]
  We present CoInteract, an end-to-end framework for HOI video synthesis conditioned on a person reference image, a product reference image, text prompts, and speech audio.<n>We propose a Human-Aware Mixture-of-Experts (MoE) that routes tokens to lightweight, region-specialized experts via spatially supervised routing.<n>Second, we propose Spatially-Structured Co-Generation, a dual-stream training paradigm that jointly models an RGB appearance stream and an auxiliary HOI structure stream to inject interaction geometry priors.
  arXiv  Detail & Related papers  (2026-04-21T16:25:43Z)
• ONE-SHOT: Compositional Human-Environment Video Synthesis via Spatial-Decoupled Motion Injection and Hybrid Context Integration [49.72976665549397]
  We propose ONE-SHOT, a parameter-efficient framework for compositional human-environment video generation.<n>Our key insight is to factorize the generative process into disentangled signals. Specifically, we introduce a canonical-space injection mechanism that decouples human dynamics from environmental cues via cross-attention.<n>Our method significantly outperforms state-of-the-art methods, offering superior structural control and creative diversity for video synthesis.
  arXiv  Detail & Related papers  (2026-04-01T15:52:00Z)
• AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation [45.753757870577196]
  We introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning.<n>We show that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses.
  arXiv  Detail & Related papers  (2026-02-04T15:42:58Z)
• EchoMotion: Unified Human Video and Motion Generation via Dual-Modality Diffusion Transformer [64.69014756863331]
  We introduce EchoMotion, a framework designed to model the joint distribution of appearance and human motion.<n>We also propose MVS-RoPE, which offers unified 3D positional encoding for both video and motion tokens.<n>Our findings reveal that explicitly representing human motion is to appearance, significantly boosting the coherence and plausibility of human-centric video generation.
  arXiv  Detail & Related papers  (2025-12-21T17:08:14Z)
• Dynamic Avatar-Scene Rendering from Human-centric Context [75.95641456716373]
  We propose bf Separate-then-Map (StM) strategy to bridge separately defined and optimized models.<n>StM significantly outperforms existing state-of-the-art methods in both visual quality and rendering accuracy.
  arXiv  Detail & Related papers  (2025-11-13T17:39:06Z)
• HumanGenesis: Agent-Based Geometric and Generative Modeling for Synthetic Human Dynamics [60.737929335600015]
  We present textbfHumanGenesis, a framework that integrates geometric and generative modeling through four collaborative agents.<n>HumanGenesis achieves state-of-the-art performance on tasks including text-guided synthesis, video reenactment, and novel-pose generalization.
  arXiv  Detail & Related papers  (2025-08-13T14:50:19Z)
• SpikeGen: Decoupled "Rods and Cones" Visual Representation Processing with Latent Generative Framework [53.27177454390712]
  This study seeks to emulate the human visual system by integrating multi-modal visual inputs with modern latent-space generative frameworks.<n>We name it SpikeGen. We evaluate its performance across various spike-RGB tasks, including conditional image and video deblurring, dense frame reconstruction from spike streams, and high-speed scene novel-view synthesis.
  arXiv  Detail & Related papers  (2025-05-23T15:54:11Z)
• AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars [71.90109867684025]
  Whole-body audio-driven avatar pose and expression generation is a critical task for creating lifelike digital humans.<n>We propose AsynFusion, a novel framework that leverages diffusion transformers to achieve cohesive expression and gesture synthesis.<n>AsynFusion achieves state-of-the-art performance in generating real-time, synchronized whole-body animations.
  arXiv  Detail & Related papers  (2025-05-21T03:28:53Z)
• JointTuner: Appearance-Motion Adaptive Joint Training for Customized Video Generation [13.168628936598367]
  JointTuner is a framework that enables joint optimization of appearance and motion components.<n>AiT Loss disrupts the flow of appearance-related components, guiding the model to focus exclusively on motion learning.<n>JointTuner is compatible with both UNet-based models and Diffusion Transformer-based models.
  arXiv  Detail & Related papers  (2025-03-31T11:04:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.