Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation

• URL: http://arxiv.org/abs/2512.11792v1
• Date: Fri, 12 Dec 2025 18:56:35 GMT
• Title: Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation
• Abstract summary: We introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX)<n>Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains.
• Score: 76.04880323498598
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reality is a dance between rigid constraints and deformable structures. For video models, that means generating motion that preserves fidelity as well as structure. Despite progress in diffusion models, producing realistic structure-preserving motion remains challenging, especially for articulated and deformable objects such as humans and animals. Scaling training data alone, so far, has failed to resolve physically implausible transitions. Existing approaches rely on conditioning with noisy motion representations, such as optical flow or skeletons extracted using an external imperfect model. To address these challenges, we introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX). With our method, we train SAM2VideoX, which contains two innovations: (1) a bidirectional feature fusion module that extracts global structure-preserving motion priors from a recurrent model like SAM2; (2) a Local Gram Flow loss that aligns how local features move together. Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains (+2.60\% on VBench, 21-22\% lower FVD, and 71.4\% human preference) over prior baselines. Specifically, on VBench, we achieve 95.51\%, surpassing REPA (92.91\%) by 2.60\%, and reduce FVD to 360.57, a 21.20\% and 22.46\% improvement over REPA- and LoRA-finetuning, respectively. The project website can be found at https://sam2videox.github.io/ .

Related papers

• Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision [78.04385989413382]
  Video diffusion models generate visually compelling content but routinely violate elementary physics when the subject involves fluids.<n>We build a physics-simulation fluid dataset combining 1,638 MPM-simulated pouring/sloshing videos with 2,320 keyword-filtered real pouring videos.<n>We introduce a dual-stream image-to-video architecture built on a pretrained diffusion-filtereder video generator.
  arXiv  Detail & Related papers  (2026-07-28T06:13:18Z)
• MobileWan: Closing the Quality Gap for Mobile Video Diffusion [53.20920841863923]
  MobileWan is the first 5B-scale video diffusion model deployable on a commercial mobile device.<n>Our system generates 5-second 480x832 videos at 16 FPS in 20 seconds end-to-end latency, achieving a VBench score of 83.79.
  arXiv  Detail & Related papers  (2026-07-07T11:52:46Z)
• BiWM: Advancing Open-Source Interactive Video World Models with Bidirectional Autoregression [13.648620674803079]
  We present BiWM, the first full-stack framework for interactive video world models under the bidirectional autoregressive paradigm.<n>From a pretrained video backbone, BiWM injects camera control by fine-tuning, then runs a few-step Distribution Matching Distillation stage.<n>A single recipe spans Wan2.1-1.3B, Wan2.2-5B, HunyuanVideo-1.5-8B, and LTX-2.3-22B, and also supports secondary fine-tuning of existing bidirectional models.
  arXiv  Detail & Related papers  (2026-06-08T20:08:41Z)
• Masked Modeling for Human Motion Recovery Under Occlusions [21.05382087890133]
  MoRo is an end-to-end generative framework that formulates motion reconstruction as a video-conditioned task.<n>MoRo achieves real-time inference at 70 FPS on a single H200 GPU.
  arXiv  Detail & Related papers  (2026-01-22T16:22:20Z)
• MAD: Motion Appearance Decoupling for efficient Driving World Models [94.40548866741791]
  We propose an efficient adaptation framework that converts generalist video models into controllable driving world models.<n>Key idea is to decouple motion learning from appearance synthesis.<n>Scaling to LTX, our MAD-LTX model outperforms all open-source competitors.
  arXiv  Detail & Related papers  (2026-01-14T12:52:23Z)
• Real-Time Motion-Controllable Autoregressive Video Diffusion [79.32730467857535]
  We propose AR-Drag, the first RL-enhanced few-step AR video diffusion model for real-time image-to-video generation with diverse motion control.<n>We first fine-tune a base I2V model to support basic motion control, then further improve it via reinforcement with a trajectory-based reward model.<n>Our design preserves the Markov property through a Self-Rollout learning mechanism and accelerates training by selectively denoising steps.
  arXiv  Detail & Related papers  (2025-10-09T12:17:11Z)
• PUSA V1.0: Surpassing Wan-I2V with $500 Training Cost by Vectorized Timestep Adaptation [18.2095668161519]
  Pusa is a groundbreaking paradigm that enables fine-grained temporal control within a unified video diffusion framework.<n>We set a new standard for image-to-video (I2V) generation, achieving a VBench-I2V total score of 87.32%.<n>This work establishes a scalable, efficient, and versatile paradigm for next-generation video synthesis.
  arXiv  Detail & Related papers  (2025-07-22T00:09:37Z)
• Physics-Guided Motion Loss for Video Generation Model [8.083315267770255]
  Current video diffusion models generate visually compelling content but often violate basic laws of physics.<n>We introduce a frequency-domain physics prior that improves motion plausibility without modifying model architectures.
  arXiv  Detail & Related papers  (2025-06-02T20:42:54Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [48.35054927704544]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• SimDA: Simple Diffusion Adapter for Efficient Video Generation [102.90154301044095]
  We propose a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B parameters of a strong T2I model, adapting it to video generation in a parameter-efficient way. In addition to T2V generation in the wild, SimDA could also be utilized in one-shot video editing with only 2 minutes tuning.
  arXiv  Detail & Related papers  (2023-08-18T17:58:44Z)
• Learning to Segment Rigid Motions from Two Frames [72.14906744113125]
  We propose a modular network, motivated by a geometric analysis of what independent object motions can be recovered from an egomotion field. It takes two consecutive frames as input and predicts segmentation masks for the background and multiple rigidly moving objects, which are then parameterized by 3D rigid transformations. Our method achieves state-of-the-art performance for rigid motion segmentation on KITTI and Sintel.
  arXiv  Detail & Related papers  (2021-01-11T04:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.