MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance

• URL: http://arxiv.org/abs/2503.16421v1
• Date: Thu, 20 Mar 2025 17:59:42 GMT
• Title: MagicMotion: Controllable Video Generation with Dense-to-Sparse Trajectory Guidance
• Authors: Quanhao Li, Zhen Xing, Rui Wang, Hui Zhang, Qi Dai, Zuxuan Wu,
• Abstract summary: We introduce MagicMotion, an image-to-video generation framework for trajectory-controllable video generation.<n>MagicMotion animates objects along defined trajectories while maintaining object consistency and visual quality.<n>We present MagicData, a large-scale trajectory-controlled video dataset, along with an automated pipeline for annotation and filtering.
• Score: 46.92591065065018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in video generation have led to remarkable improvements in visual quality and temporal coherence. Upon this, trajectory-controllable video generation has emerged to enable precise object motion control through explicitly defined spatial paths. However, existing methods struggle with complex object movements and multi-object motion control, resulting in imprecise trajectory adherence, poor object consistency, and compromised visual quality. Furthermore, these methods only support trajectory control in a single format, limiting their applicability in diverse scenarios. Additionally, there is no publicly available dataset or benchmark specifically tailored for trajectory-controllable video generation, hindering robust training and systematic evaluation. To address these challenges, we introduce MagicMotion, a novel image-to-video generation framework that enables trajectory control through three levels of conditions from dense to sparse: masks, bounding boxes, and sparse boxes. Given an input image and trajectories, MagicMotion seamlessly animates objects along defined trajectories while maintaining object consistency and visual quality. Furthermore, we present MagicData, a large-scale trajectory-controlled video dataset, along with an automated pipeline for annotation and filtering. We also introduce MagicBench, a comprehensive benchmark that assesses both video quality and trajectory control accuracy across different numbers of objects. Extensive experiments demonstrate that MagicMotion outperforms previous methods across various metrics. Our project page are publicly available at https://quanhaol.github.io/magicmotion-site.

Related papers

• PoseTraj: Pose-Aware Trajectory Control in Video Diffusion [17.0187150041712]
  We introduce PoseTraj, a pose-aware video dragging model for generating 3D-aligned motion from 2D trajectories.<n>Our method adopts a novel two-stage pose-aware pretraining framework, improving 3D understanding across diverse trajectories.
  arXiv  Detail & Related papers  (2025-03-20T12:01:43Z)
• C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation [81.4106601222722]
  Trajectory-based motion control has emerged as an intuitive and efficient approach for controllable video generation.<n>We propose a Chain-of-Thought-based motion controller for controllable video generation, named C-Drag.<n>Our method includes an object perception module and a Chain-of-Thought-based motion reasoning module.
  arXiv  Detail & Related papers  (2025-02-27T08:21:03Z)
• 3DTrajMaster: Mastering 3D Trajectory for Multi-Entity Motion in Video Generation [83.98251722144195]
  Previous methods on controllable video generation primarily leverage 2D control signals to manipulate object motions.<n>We introduce 3DTrajMaster, a robust controller that regulates multi-entity dynamics in 3D space.<n>We show that 3DTrajMaster sets a new state-of-the-art in both accuracy and generalization for controlling multi-entity 3D motions.
  arXiv  Detail & Related papers  (2024-12-10T18:55:13Z)
• Motion Prompting: Controlling Video Generation with Motion Trajectories [57.049252242807874]
  We train a video generation model conditioned on sparse or dense video trajectories.<n>We translate high-level user requests into detailed, semi-dense motion prompts.<n>We demonstrate our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing.
  arXiv  Detail & Related papers  (2024-12-03T18:59:56Z)
• TrackGo: A Flexible and Efficient Method for Controllable Video Generation [33.62804888664707]
  We introduce TrackGo, a novel approach that leverages free-form masks and arrows for conditional video generation.<n>We also propose the TrackAdapter for control implementation, an efficient and lightweight adapter designed to be seamlessly integrated into the temporal self-attention layers.<n>Our experimental results demonstrate that our new approach, enhanced by the TrackAdapter, achieves state-of-the-art performance on key metrics such as FVD, FID, and MC scores.
  arXiv  Detail & Related papers  (2024-08-21T09:42:04Z)
• MotionBooth: Motion-Aware Customized Text-to-Video Generation [44.41894050494623]
  MotionBooth is a framework designed for animating customized subjects with precise control over both object and camera movements. We efficiently fine-tune a text-to-video model to capture the object's shape and attributes accurately. Our approach presents subject region loss and video preservation loss to enhance the subject's learning performance.
  arXiv  Detail & Related papers  (2024-06-25T17:42:25Z)
• Image Conductor: Precision Control for Interactive Video Synthesis [90.2353794019393]
  Filmmaking and animation production often require sophisticated techniques for coordinating camera transitions and object movements. Image Conductor is a method for precise control of camera transitions and object movements to generate video assets from a single image.
  arXiv  Detail & Related papers  (2024-06-21T17:55:05Z)
• Ctrl-V: Higher Fidelity Video Generation with Bounding-Box Controlled Object Motion [8.068194154084967]
  This paper tackles a challenge of how to exert precise control over object motion for realistic video synthesis. To accomplish this, we control object movements using bounding boxes and extend this control to the renderings of 2D or 3D boxes in pixel space. Our method, Ctrl-V, leverages modified and fine-tuned Stable Video Diffusion (SVD) models to solve both trajectory and video generation.
  arXiv  Detail & Related papers  (2024-06-09T03:44:35Z)
• DragNUWA: Fine-grained Control in Video Generation by Integrating Text, Image, and Trajectory [126.4597063554213]
  DragNUWA is an open-domain diffusion-based video generation model. It provides fine-grained control over video content from semantic, spatial, and temporal perspectives. Our experiments validate the effectiveness of DragNUWA, demonstrating its superior performance in fine-grained control in video generation.
  arXiv  Detail & Related papers  (2023-08-16T01:43:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.