LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation

• URL: http://arxiv.org/abs/2605.23878v1
• Date: Fri, 22 May 2026 17:34:42 GMT
• Title: LaMo: Self-Supervised Latent Motion Priors for Physical Realism in Video Generation
• Abstract summary: We propose LaMo, which formulates a latent motion prior over frame-to-frame latent changes conditioned on the current latent and prompt.<n>LaMo is plug-and-play with existing video diffusion backbones, requiring no architectural or I/O changes.<n>On VideoPhy and VideoPhy2, LaMo improves CogVideoX backbones and outperforms recent physics-aware baselines that use external supervision.
• Score: 24.8120698643545
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern video generators produce visually compelling clips but still struggle with physical and motion consistency, limiting their use as reliable world simulators. Existing remedies often rely on external simulators, teacher models, or curated physics-focused data. We explore a complementary self-supervised direction: extracting motion cues from the unlabeled videos already used to train video diffusion models. We propose LaMo, which formulates a latent motion prior over frame-to-frame latent changes conditioned on the current latent and prompt. This prior is exposed through two lightweight readouts: a macro motion drift used during training as a Motion Drift Loss, and a learned micro motion field used during sampling as Motion Prior Guidance. Both components are plug-and-play with existing video diffusion backbones, requiring no architectural or I/O changes. On VideoPhy and VideoPhy2, LaMo improves CogVideoX backbones and outperforms recent physics-aware baselines that use external supervision. On VBench, it preserves overall generation quality while improving motion-related dimensions. These results suggest that unlabeled video contains useful motion supervision for improving physical fidelity in modern video diffusion models.

Related papers

• SIFT: Self-Imagination Fine-Tuning for Physically Plausible Motion in Video Diffusion Models [53.82339966194578]
  Motion entanglement is the unintended coupling of independent motion sources, such as camera movement and object motion.<n>We propose a Self-Imagination Fine-Tuning (SIFT) paradigm, which enables the model to learn from its own generated videos.
  arXiv  Detail & Related papers  (2026-06-26T05:39:40Z)
• MoAlign: Motion-Centric Representation Alignment for Video Diffusion Models [50.162882483045045]
  We propose a motion-centric alignment framework that learns a disentangled motion subspace from a pretrained video encoder.<n>This subspace is optimized to predict ground-truth optical flow, ensuring it captures true motion dynamics.<n>We then align the latent features of a text-to-video diffusion model to this new subspace, enabling the generative model to internalize motion knowledge and generate more plausible videos.
  arXiv  Detail & Related papers  (2025-10-21T19:05:23Z)
• Motion Prompting: Controlling Video Generation with Motion Trajectories [57.049252242807874]
  We train a video generation model conditioned on sparse or dense video trajectories.<n>We translate high-level user requests into detailed, semi-dense motion prompts.<n>We demonstrate our approach through various applications, including camera and object motion control, "interacting" with an image, motion transfer, and image editing.
  arXiv  Detail & Related papers  (2024-12-03T18:59:56Z)
• MotionCraft: Physics-based Zero-Shot Video Generation [22.33113030344355]
  MotionCraft is a new zero-shot video generator to craft physics-based and realistic videos. We show that MotionCraft is able to warp the noise latent space of an image diffusion model, such as Stable Diffusion, by applying an optical flow. We compare our method with the state-of-the-art Text2Video-Zero reporting qualitative and quantitative improvements.
  arXiv  Detail & Related papers  (2024-05-22T11:44:57Z)
• Spectral Motion Alignment for Video Motion Transfer using Diffusion Models [54.32923808964701]
  Spectral Motion Alignment (SMA) is a framework that refines and aligns motion vectors using Fourier and wavelet transforms.<n> SMA learns motion patterns by incorporating frequency-domain regularization, facilitating the learning of whole-frame global motion dynamics.<n>Extensive experiments demonstrate SMA's efficacy in improving motion transfer while maintaining computational efficiency and compatibility across various video customization frameworks.
  arXiv  Detail & Related papers  (2024-03-22T14:47:18Z)
• MotionCrafter: One-Shot Motion Customization of Diffusion Models [66.44642854791807]
  We introduce MotionCrafter, a one-shot instance-guided motion customization method. MotionCrafter employs a parallel spatial-temporal architecture that injects the reference motion into the temporal component of the base model. During training, a frozen base model provides appearance normalization, effectively separating appearance from motion.
  arXiv  Detail & Related papers  (2023-12-08T16:31:04Z)
• VMC: Video Motion Customization using Temporal Attention Adaption for Text-to-Video Diffusion Models [58.93124686141781]
  Video Motion Customization (VMC) is a novel one-shot tuning approach crafted to adapt temporal attention layers within video diffusion models. Our approach introduces a novel motion distillation objective using residual vectors between consecutive frames as a motion reference. We validate our method against state-of-the-art video generative models across diverse real-world motions and contexts.
  arXiv  Detail & Related papers  (2023-12-01T06:50:11Z)
• LaMD: Latent Motion Diffusion for Image-Conditional Video Generation [63.34574080016687]
  latent motion diffusion (LaMD) framework consists of a motion-decomposed video autoencoder and a diffusion-based motion generator.<n>LaMD generates high-quality videos on various benchmark datasets, including BAIR, Landscape, NATOPS, MUG and CATER-GEN.
  arXiv  Detail & Related papers  (2023-04-23T10:32:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.