MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models

• URL: http://arxiv.org/abs/2510.19022v1
• Date: Tue, 21 Oct 2025 19:05:23 GMT
• Title: MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models
• Authors: Aritra Bhowmik, Denis Korzhenkov, Cees G. M. Snoek, Amirhossein Habibian, Mohsen Ghafoorian,
• Abstract summary: We propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder.<n>This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics.<n>We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos.
• Score: 50.162882483045045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-video diffusion models have enabled high-quality video synthesis, yet often fail to generate temporally coherent and physically plausible motion. A key reason is the models' insufficient understanding of complex motions that natural videos often entail. Recent works tackle this problem by aligning diffusion model features with those from pretrained video encoders. However, these encoders mix video appearance and dynamics into entangled features, limiting the benefit of such alignment. In this paper, we propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder. This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics. We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos. Our method improves the physical commonsense in a state-of-the-art video diffusion model, while preserving adherence to textual prompts, as evidenced by empirical evaluations on VideoPhy, VideoPhy2, VBench, and VBench-2.0, along with a user study.

Related papers

• Moaw: Unleashing Motion Awareness for Video Diffusion Models [71.34328578845721]
  Moaw is a framework that unleashes motion awareness for video diffusion models.<n>We train a diffusion model for motion perception, shifting its modality from image-to-video generation to video-to-dense-tracking.<n>We then construct a motion-labeled dataset to identify features that encode the strongest motion information, and inject them into a structurally identical video generation model.
  arXiv  Detail & Related papers  (2026-01-19T06:45:46Z)
• VideoPhy: Evaluating Physical Commonsense for Video Generation [93.28748850301949]
  We present VideoPhy, a benchmark designed to assess whether the generated videos follow physical commonsense for real-world activities. We then generate videos conditioned on captions from diverse state-of-the-art text-to-video generative models. Our human evaluation reveals that the existing models severely lack the ability to generate videos adhering to the given text prompts.
  arXiv  Detail & Related papers  (2024-06-05T17:53:55Z)
• MotionCraft: Physics-based Zero-Shot Video Generation [22.33113030344355]
  MotionCraft is a new zero-shot video generator to craft physics-based and realistic videos. We show that MotionCraft is able to warp the noise latent space of an image diffusion model, such as Stable Diffusion, by applying an optical flow. We compare our method with the state-of-the-art Text2Video-Zero reporting qualitative and quantitative improvements.
  arXiv  Detail & Related papers  (2024-05-22T11:44:57Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• MoVideo: Motion-Aware Video Generation with Diffusion Models [97.03352319694795]
  We propose a novel motion-aware generation (MoVideo) framework that takes motion into consideration from two aspects: video depth and optical flow. MoVideo achieves state-of-the-art results in both text-to-video and image-to-video generation, showing promising prompt consistency, frame consistency and visual quality.
  arXiv  Detail & Related papers  (2023-11-19T13:36:03Z)
• LLM-grounded Video Diffusion Models [57.23066793349706]
  Video diffusion models have emerged as a promising tool for neuraltemporal generation. Current models struggle with prompts and often restricted or incorrect motion. We introduce LLM-grounded Video Diffusion (LVD) Our results demonstrate that LVD significantly outperforms its base video diffusion model.
  arXiv  Detail & Related papers  (2023-09-29T17:54:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.