I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength

• URL: http://arxiv.org/abs/2411.06525v2
• Date: Tue, 26 Nov 2024 03:53:29 GMT
• Title: I2VControl-Camera: Precise Video Camera Control with Adjustable Motion Strength
• Authors: Wanquan Feng, Jiawei Liu, Pengqi Tu, Tianhao Qi, Mingzhen Sun, Tianxiang Ma, Songtao Zhao, Siyu Zhou, Qian He,
• Abstract summary: I2VControl-Camera is a novel camera control method that significantly enhances controllability while providing over the strength of subject motion. To accurately control and adjust the strength of subject motion, we explicitly model the higher-order components of the video trajectory expansion.
• Score: 11.778832811404259
• License:
• Abstract: Video generation technologies are developing rapidly and have broad potential applications. Among these technologies, camera control is crucial for generating professional-quality videos that accurately meet user expectations. However, existing camera control methods still suffer from several limitations, including control precision and the neglect of the control for subject motion dynamics. In this work, we propose I2VControl-Camera, a novel camera control method that significantly enhances controllability while providing adjustability over the strength of subject motion. To improve control precision, we employ point trajectory in the camera coordinate system instead of only extrinsic matrix information as our control signal. To accurately control and adjust the strength of subject motion, we explicitly model the higher-order components of the video trajectory expansion, not merely the linear terms, and design an operator that effectively represents the motion strength. We use an adapter architecture that is independent of the base model structure. Experiments on static and dynamic scenes show that our framework outperformances previous methods both quantitatively and qualitatively. The project page is: https://wanquanf.github.io/I2VControlCamera .

Related papers

• CamI2V: Camera-Controlled Image-to-Video Diffusion Model [11.762824216082508]
  In this paper, we emphasize the necessity of integrating explicit physical constraints into model design. Epipolar attention is proposed for modeling all cross-frame relationships from a novel perspective of noised condition. We achieve a 25.5% improvement in camera controllability on RealEstate10K while maintaining strong generalization to out-of-domain images.
  arXiv  Detail & Related papers  (2024-10-21T12:36:27Z)
• Boosting Camera Motion Control for Video Diffusion Transformers [21.151900688555624]
  We show that transformer-based diffusion models (DiT) suffer from severe degradation in camera motion accuracy. To address the persistent motion degradation in DiT, we introduce Camera Motion Guidance (CMG), which boosts camera control by over 400%. Our method universally applies to both U-Net and DiT models, offering improved camera control for video generation tasks.
  arXiv  Detail & Related papers  (2024-10-14T17:58:07Z)
• VD3D: Taming Large Video Diffusion Transformers for 3D Camera Control [74.5434726968562]
  We tame transformers video for 3D camera control using a ControlNet-like conditioning mechanism based on Plucker coordinates. Our work is the first to enable camera control for transformer-based video diffusion models.
  arXiv  Detail & Related papers  (2024-07-17T17:59:05Z)
• Image Conductor: Precision Control for Interactive Video Synthesis [90.2353794019393]
  Filmmaking and animation production often require sophisticated techniques for coordinating camera transitions and object movements. Image Conductor is a method for precise control of camera transitions and object movements to generate video assets from a single image.
  arXiv  Detail & Related papers  (2024-06-21T17:55:05Z)
• MotionMaster: Training-free Camera Motion Transfer For Video Generation [48.706578330771386]
  We propose a novel training-free video motion transfer model, which disentangles camera motions and object motions in source videos. Our model can effectively decouple camera-object motion and apply the decoupled camera motion to a wide range of controllable video generation tasks.
  arXiv  Detail & Related papers  (2024-04-24T10:28:54Z)
• CameraCtrl: Enabling Camera Control for Text-to-Video Generation [86.36135895375425]
  Controllability plays a crucial role in video generation since it allows users to create desired content. Existing models largely overlooked the precise control of camera pose that serves as a cinematic language. We introduce CameraCtrl, enabling accurate camera pose control for text-to-video(T2V) models.
  arXiv  Detail & Related papers  (2024-04-02T16:52:41Z)
• Direct-a-Video: Customized Video Generation with User-Directed Camera Movement and Object Motion [34.404342332033636]
  We introduce Direct-a-Video, a system that allows users to independently specify motions for multiple objects as well as camera's pan and zoom movements. For camera movement, we introduce new temporal cross-attention layers to interpret quantitative camera movement parameters. Both components operate independently, allowing individual or combined control, and can generalize to open-domain scenarios.
  arXiv  Detail & Related papers  (2024-02-05T16:30:57Z)
• MotionCtrl: A Unified and Flexible Motion Controller for Video Generation [77.09621778348733]
  Motions in a video primarily consist of camera motion, induced by camera movement, and object motion, resulting from object movement. This paper presents MotionCtrl, a unified motion controller for video generation designed to effectively and independently control camera and object motion.
  arXiv  Detail & Related papers  (2023-12-06T17:49:57Z)
• Fine-grained Controllable Video Generation via Object Appearance and Context [74.23066823064575]
  We propose fine-grained controllable video generation (FACTOR) to achieve detailed control. FACTOR aims to control objects' appearances and context, including their location and category. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users.
  arXiv  Detail & Related papers  (2023-12-05T17:47:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.