MotiMotion: Motion-Controlled Video Generation with Visual Reasoning

• URL: http://arxiv.org/abs/2605.22818v1
• Date: Thu, 21 May 2026 17:59:36 GMT
• Title: MotiMotion: Motion-Controlled Video Generation with Visual Reasoning
• Abstract summary: We introduce MotiMotion, a framework that reformulates motion control as a reasoning-then-generation problem.<n>To encourage causally grounded and commonsense-consistent interactions, we leverage a training-free vision-language reasoner.<n>To further improve motion naturalness, we propose a confidence-aware control scheme that modulates guidance strength.
• Score: 49.243234645532745
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current motion-controlled image-to-video generation models rigidly follow user-provided trajectories that are often sparse, imprecise, and causally incomplete. Such reliance often yields unnatural or implausible outcomes, especially by missing secondary causal consequences. To address this, we introduce MotiMotion, a novel framework that reformulates motion control as a reasoning-then-generation problem. To encourage causally grounded and commonsense-consistent interactions, we leverage a training-free vision-language reasoner to refine image-space coordinates of primary trajectories and to hallucinate plausible secondary motions. To further improve motion naturalness, we propose a confidence-aware control scheme that modulates guidance strength, enabling the model to closely follow high-confidence plans while correcting artifacts under low-confidence inputs with its internal generative priors. To support systematic evaluation, we curate a new image-to-video benchmark, MotiBench, consisting of interaction-centric scenes where new events are triggered by motion. Both VLM-based evaluation and a human study on MotiBench demonstrate that MotiMotion produces videos with more plausible object behaviors and interaction, and is preferred over existing approaches.

Related papers

• SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models [53.82339966194578]
  Motion entanglement is the unintended coupling of independent motion sources, such as camera movement and object motion.<n>We propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos.
  arXiv  Detail & Related papers  (2026-06-26T05:39:40Z)
• MotionMERGE: A Multi-granular Framework for Human Motion Editing, Reasoning, Generation, and Explanation [66.66098171359995]
  MotionMERGE is a unified framework that bridges the granularity gap in motion-language models.<n>First, we pioneer the study of fine-grained language-guided motion control, including detailed understanding and localized editing.<n>Second, we design ReasoningAware Granularity-Synergy pre-training, a novel strategy that employs joint supervision for cross-granularity alignment.<n>Third, we curate MotionFineEdit, a large-scale dataset with the first finegrained-temporal corrective instructions and motion-grounded CoT annotations.
  arXiv  Detail & Related papers  (2026-05-18T18:00:04Z)
• EgoMotion: Hierarchical Reasoning and Diffusion for Egocentric Vision-Language Motion Generation [74.07852250099559]
  We investigate Egocentric Vision-Language (Ego-VL) motion generation.<n>This task requires synthesizing 3D human motion conditioned jointly on first-person visual observations and natural language instructions.<n>We propose a hierarchical generative framework textbfEgoMotion to address this challenge.
  arXiv  Detail & Related papers  (2026-04-21T05:31:06Z)
• TokenMotion: Decoupled Motion Control via Token Disentanglement for Human-centric Video Generation [7.900728371180723]
  We present TokenMotion, the first DiT-based video diffusion framework that enables fine-grained control over camera motion.<n>Our approach introduces a unified modeling framework utilizing a decouple-and-fuse strategy, bridged by a human-aware dynamic mask.<n>Our work represents a significant advancement in controllable video generation, with particular relevance for creative production applications.
  arXiv  Detail & Related papers  (2025-04-11T00:41:25Z)
• VideoJAM: Joint Appearance-Motion Representations for Enhanced Motion Generation in Video Models [71.9811050853964]
  VideoJAM is a novel framework that instills an effective motion prior to video generators.<n>VideoJAM achieves state-of-the-art performance in motion coherence.<n>These findings emphasize that appearance and motion can be complementary and, when effectively integrated, enhance both the visual quality and the coherence of video generation.
  arXiv  Detail & Related papers  (2025-02-04T17:07:10Z)
• Motion-Aware Generative Frame Interpolation [23.380470636851022]
  Flow-based frame methods ensure motion stability through estimated intermediate flow but often introduce severe artifacts in complex motion regions.<n>Recent generative approaches, boosted by large-scale pre-trained video generation models, show promise in handling intricate scenes.<n>We propose Motion-aware Generative frame (MoG) that synergizes intermediate flow guidance with generative capacities to enhance fidelity.
  arXiv  Detail & Related papers  (2025-01-07T11:03:43Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• MotionClone: Training-Free Motion Cloning for Controllable Video Generation [41.621147782128396]
  MotionClone is a training-free framework that enables motion cloning from reference videos to versatile motion-controlled video generation. MotionClone exhibits proficiency in both global camera motion and local object motion, with notable superiority in terms of motion fidelity, textual alignment, and temporal consistency.
  arXiv  Detail & Related papers  (2024-06-08T03:44:25Z)
• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms.<n> SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics.<n>Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• Learning Variational Motion Prior for Video-based Motion Capture [31.79649766268877]
  We present a novel variational motion prior (VMP) learning approach for video-based motion capture. Our framework can effectively reduce temporal jittering and failure modes in frame-wise pose estimation. Experiments over both public datasets and in-the-wild videos have demonstrated the efficacy and generalization capability of our framework.
  arXiv  Detail & Related papers  (2022-10-27T02:45:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.