SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models

• URL: http://arxiv.org/abs/2606.27741v1
• Date: Fri, 26 Jun 2026 05:39:40 GMT
• Title: SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models
• Abstract summary: Motion entanglement is the unintended coupling of independent motion sources, such as camera movement and object motion.<n>We propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos.
• Score: 53.82339966194578
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in video diffusion models have greatly improved visual fidelity, yet their generated motions often violate physical plausibility. We observe a common kinematic failure, "motion entanglement", the unintended coupling of independent motion sources, such as camera movement and object motion. We identify that this issue stems from data bias and the reconstruction-based training design of diffusion models. Training on noisy videos that still retain coarse motion cues inadvertently encourages the model to replicate existing motion without an incentive to learn how to model kinematically-grounded motions. To address this, we propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos rather than directly reconstructing real ones, breaking the reconstruction shortcut. We further employ motion-aware discriminative supervision and a progressive hard-case replay strategy to stabilize and accelerate learning. By leveraging freely-generated text prompts, our method can densely cover a broad motion space, including rare or finely-disentangled scenarios that would be costly to collect as video data. Extensive experiments demonstrate that our approach substantially improves the physical realism, motion disentanglement, and controllability of generated videos.

Related papers

• LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation [24.8120698643545]
  We propose LaMo, which formulates a latent motion prior over frame-to-frame latent changes conditioned on the current latent and prompt.<n>LaMo is plug-and-play with existing video diffusion backbones, requiring no architectural or I/O changes.<n>On VideoPhy and VideoPhy2, LaMo improves CogVideoX backbones and outperforms recent physics-aware baselines that use external supervision.
  arXiv  Detail & Related papers  (2026-05-22T17:34:42Z)
• MotiMotion: Motion-Controlled Video Generation with Visual Reasoning [49.243234645532745]
  We introduce MotiMotion, a framework that reformulates motion control as a reasoning-then-generation problem.<n>To encourage causally grounded and commonsense-consistent interactions, we leverage a training-free vision-language reasoner.<n>To further improve motion naturalness, we propose a confidence-aware control scheme that modulates guidance strength.
  arXiv  Detail & Related papers  (2026-05-21T17:59:36Z)
• DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data [51.316274891736164]
  DynaVid is a video synthesis framework that leverages synthetic motion data in training.<n>We show that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.
  arXiv  Detail & Related papers  (2026-04-02T06:12:38Z)
• Moaw: Unleashing Motion Awareness for Video Diffusion Models [71.34328578845721]
  Moaw is a framework that unleashes motion awareness for video diffusion models.<n>We train a diffusion model for motion perception, shifting its modality from image-to-video generation to video-to-dense-tracking.<n>We then construct a motion-labeled dataset to identify features that encode the strongest motion information, and inject them into a structurally identical video generation model.
  arXiv  Detail & Related papers  (2026-01-19T06:45:46Z)
• RAGME: Retrieval Augmented Video Generation for Enhanced Motion Realism [73.38167494118746]
  We propose a framework to improve the realism of motion in generated videos.<n>We advocate for the incorporation of a retrieval mechanism during the generation phase.<n>Our pipeline is designed to apply to any text-to-video diffusion model.
  arXiv  Detail & Related papers  (2025-04-09T08:14:05Z)
• MotionFlow: Attention-Driven Motion Transfer in Video Diffusion Models [3.2311303453753033]
  We introduce MotionFlow, a novel framework designed for motion transfer in video diffusion models.<n>Our method utilizes cross-attention maps to accurately capture and manipulate spatial and temporal dynamics.<n>Our experiments demonstrate that MotionFlow significantly outperforms existing models in both fidelity and versatility even during drastic scene alterations.
  arXiv  Detail & Related papers  (2024-12-06T18:59:12Z)
• MoTrans: Customized Motion Transfer with Text-driven Video Diffusion Models [59.10171699717122]
  MoTrans is a customized motion transfer method enabling video generation of similar motion in new context.<n> multimodal representations from recaptioned prompt and video frames promote the modeling of appearance.<n>Our method effectively learns specific motion pattern from singular or multiple reference videos.
  arXiv  Detail & Related papers  (2024-12-02T10:07:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.