DreamCinema: Cinematic Transfer with Free Camera and 3D Character

• URL: http://arxiv.org/abs/2408.12601v1
• Date: Thu, 22 Aug 2024 17:59:44 GMT
• Title: DreamCinema: Cinematic Transfer with Free Camera and 3D Character
• Authors: Weiliang Chen, Fangfu Liu, Diankun Wu, Haowen Sun, Haixu Song, Yueqi Duan,
• Abstract summary: We propose DreamCinema, a novel cinematic transfer framework that pioneers generative AI into the film production paradigm. Specifically, we first extract cinematic elements (i.e., human and camera pose) and optimize the camera trajectory. Then, we apply a character generator to efficiently create 3D high-quality characters with a human structure prior.
• Score: 11.979669977372707
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We are living in a flourishing era of digital media, where everyone has the potential to become a personal filmmaker. Current research on cinematic transfer empowers filmmakers to reproduce and manipulate the visual elements (e.g., cinematography and character behaviors) from classic shots. However, characters in the reimagined films still rely on manual crafting, which involves significant technical complexity and high costs, making it unattainable for ordinary users. Furthermore, their estimated cinematography lacks smoothness due to inadequate capturing of inter-frame motion and modeling of physical trajectories. Fortunately, the remarkable success of 2D and 3D AIGC has opened up the possibility of efficiently generating characters tailored to users' needs, diversifying cinematography. In this paper, we propose DreamCinema, a novel cinematic transfer framework that pioneers generative AI into the film production paradigm, aiming at facilitating user-friendly film creation. Specifically, we first extract cinematic elements (i.e., human and camera pose) and optimize the camera trajectory. Then, we apply a character generator to efficiently create 3D high-quality characters with a human structure prior. Finally, we develop a structure-guided motion transfer strategy to incorporate generated characters into film creation and transfer it via 3D graphics engines smoothly. Extensive experiments demonstrate the effectiveness of our method for creating high-quality films with free camera and 3D characters.

Related papers

• Motion Diffusion-Guided 3D Global HMR from a Dynamic Camera [3.6948631725065355]
  We present DiffOpt, a novel 3D global HMR method using Diffusion Optimization. Our key insight is that recent advances in human motion generation, such as the motion diffusion model (MDM), contain a strong prior of coherent human motion. We validate DiffOpt with video sequences from the Electromagnetic Database of Global 3D Human Pose and Shape in the Wild.
  arXiv  Detail & Related papers  (2024-11-15T21:09:40Z)
• ChatCam: Empowering Camera Control through Conversational AI [67.31920821192323]
  ChatCam is a system that navigates camera movements through conversations with users. To achieve this, we propose CineGPT, a GPT-based autoregressive model for text-conditioned camera trajectory generation. We also develop an Anchor Determinator to ensure precise camera trajectory placement.
  arXiv  Detail & Related papers  (2024-09-25T20:13:41Z)
• Cinematic Behavior Transfer via NeRF-based Differentiable Filming [63.1622492808519]
  Existing SLAM methods face limitations in dynamic scenes and human pose estimation often focuses on 2D projections. We first introduce a reverse filming behavior estimation technique. We then introduce a cinematic transfer pipeline that is able to transfer various shot types to a new 2D video or a 3D virtual environment.
  arXiv  Detail & Related papers  (2023-11-29T15:56:58Z)
• Automatic Camera Trajectory Control with Enhanced Immersion for Virtual Cinematography [23.070207691087827]
  Real-world cinematographic rules show that directors can create immersion by comprehensively synchronizing the camera with the actor. Inspired by this strategy, we propose a deep camera control framework that enables actor-camera synchronization in three aspects. Our proposed method yields immersive cinematic videos of high quality, both quantitatively and qualitatively.
  arXiv  Detail & Related papers  (2023-03-29T22:02:15Z)
• AgileGAN3D: Few-Shot 3D Portrait Stylization by Augmented Transfer Learning [80.67196184480754]
  We propose a novel framework emphAgileGAN3D that can produce 3D artistically appealing portraits with detailed geometry. New stylization can be obtained with just a few (around 20) unpaired 2D exemplars. Our pipeline demonstrates strong capability in turning user photos into a diverse range of 3D artistic portraits.
  arXiv  Detail & Related papers  (2023-03-24T23:04:20Z)
• 3D Cinemagraphy from a Single Image [73.09720823592092]
  We present 3D Cinemagraphy, a new technique that marries 2D image animation with 3D photography. Given a single still image as input, our goal is to generate a video that contains both visual content animation and camera motion.
  arXiv  Detail & Related papers  (2023-03-10T06:08:23Z)
• Decoupling Human and Camera Motion from Videos in the Wild [67.39432972193929]
  We propose a method to reconstruct global human trajectories from videos in the wild. Our method decouples the camera and human motion, which allows us to place people in the same world coordinate frame.
  arXiv  Detail & Related papers  (2023-02-24T18:59:15Z)
• Action2video: Generating Videos of Human 3D Actions [31.665831044217363]
  We aim to tackle the interesting yet challenging problem of generating videos of diverse and natural human motions from prescribed action categories. Key issue lies in the ability to synthesize multiple distinct motion sequences that are realistic in their visual appearances. Action2motionally generates plausible 3D pose sequences of a prescribed action category, which are processed and rendered by motion2video to form 2D videos.
  arXiv  Detail & Related papers  (2021-11-12T20:20:37Z)
• Learning Motion Priors for 4D Human Body Capture in 3D Scenes [81.54377747405812]
  We propose LEMO: LEarning human MOtion priors for 4D human body capture. We introduce a novel motion prior, which reduces the jitters exhibited by poses recovered over a sequence. We also design a contact friction term and a contact-aware motion infiller obtained via per-instance self-supervised training. With our pipeline, we demonstrate high-quality 4D human body capture, reconstructing smooth motions and physically plausible body-scene interactions.
  arXiv  Detail & Related papers  (2021-08-23T20:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.