Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation

• URL: http://arxiv.org/abs/2512.11792v1
• Date: Fri, 12 Dec 2025 18:56:35 GMT
• Title: Structure From Tracking: Distilling Structure-Preserving Motion for Video Generation
• Authors: Yang Fei, George Stoica, Jingyuan Liu, Qifeng Chen, Ranjay Krishna, Xiaojuan Wang, Benlin Liu,
• Abstract summary: We introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX)<n>Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains.
• Score: 76.04880323498598
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Reality is a dance between rigid constraints and deformable structures. For video models, that means generating motion that preserves fidelity as well as structure. Despite progress in diffusion models, producing realistic structure-preserving motion remains challenging, especially for articulated and deformable objects such as humans and animals. Scaling training data alone, so far, has failed to resolve physically implausible transitions. Existing approaches rely on conditioning with noisy motion representations, such as optical flow or skeletons extracted using an external imperfect model. To address these challenges, we introduce an algorithm to distill structure-preserving motion priors from an autoregressive video tracking model (SAM2) into a bidirectional video diffusion model (CogVideoX). With our method, we train SAM2VideoX, which contains two innovations: (1) a bidirectional feature fusion module that extracts global structure-preserving motion priors from a recurrent model like SAM2; (2) a Local Gram Flow loss that aligns how local features move together. Experiments on VBench and in human studies show that SAM2VideoX delivers consistent gains (+2.60\% on VBench, 21-22\% lower FVD, and 71.4\% human preference) over prior baselines. Specifically, on VBench, we achieve 95.51\%, surpassing REPA (92.91\%) by 2.60\%, and reduce FVD to 360.57, a 21.20\% and 22.46\% improvement over REPA- and LoRA-finetuning, respectively. The project website can be found at https://sam2videox.github.io/ .

Related papers

• Masked Modeling for Human Motion Recovery Under Occlusions [21.05382087890133]
  MoRo is an end-to-end generative framework that formulates motion reconstruction as a video-conditioned task.<n>MoRo achieves real-time inference at 70 FPS on a single H200 GPU.
  arXiv  Detail & Related papers  (2026-01-22T16:22:20Z)
• MAD: Motion Appearance Decoupling for efficient Driving World Models [94.40548866741791]
  We propose an efficient adaptation framework that converts generalist video models into controllable driving world models.<n>Key idea is to decouple motion learning from appearance synthesis.<n>Scaling to LTX, our MAD-LTX model outperforms all open-source competitors.
  arXiv  Detail & Related papers  (2026-01-14T12:52:23Z)
• Real-Time Motion-Controllable Autoregressive Video Diffusion [79.32730467857535]
  We propose AR-Drag, the first RL-enhanced few-step AR video diffusion model for real-time image-to-video generation with diverse motion control.<n>We first fine-tune a base I2V model to support basic motion control, then further improve it via reinforcement with a trajectory-based reward model.<n>Our design preserves the Markov property through a Self-Rollout learning mechanism and accelerates training by selectively denoising steps.
  arXiv  Detail & Related papers  (2025-10-09T12:17:11Z)
• PUSA V1.0: Surpassing Wan-I2V with $500 Training Cost by Vectorized Timestep Adaptation [18.2095668161519]
  Pusa is a groundbreaking paradigm that enables fine-grained temporal control within a unified video diffusion framework.<n>We set a new standard for image-to-video (I2V) generation, achieving a VBench-I2V total score of 87.32%.<n>This work establishes a scalable, efficient, and versatile paradigm for next-generation video synthesis.
  arXiv  Detail & Related papers  (2025-07-22T00:09:37Z)
• Physics-Guided Motion Loss for Video Generation Model [8.083315267770255]
  Current video diffusion models generate visually compelling content but often violate basic laws of physics.<n>We introduce a frequency-domain physics prior that improves motion plausibility without modifying model architectures.
  arXiv  Detail & Related papers  (2025-06-02T20:42:54Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [48.35054927704544]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies.<n>We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly.<n>Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• SimDA: Simple Diffusion Adapter for Efficient Video Generation [102.90154301044095]
  We propose a Simple Diffusion Adapter (SimDA) that fine-tunes only 24M out of 1.1B parameters of a strong T2I model, adapting it to video generation in a parameter-efficient way. In addition to T2V generation in the wild, SimDA could also be utilized in one-shot video editing with only 2 minutes tuning.
  arXiv  Detail & Related papers  (2023-08-18T17:58:44Z)
• Learning to Segment Rigid Motions from Two Frames [72.14906744113125]
  We propose a modular network, motivated by a geometric analysis of what independent object motions can be recovered from an egomotion field. It takes two consecutive frames as input and predicts segmentation masks for the background and multiple rigidly moving objects, which are then parameterized by 3D rigid transformations. Our method achieves state-of-the-art performance for rigid motion segmentation on KITTI and Sintel.
  arXiv  Detail & Related papers  (2021-01-11T04:20:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.