Self-Supervised Learning of Structured Dynamics from Videos

• URL: http://arxiv.org/abs/2607.21576v1
• Date: Thu, 23 Jul 2026 17:55:07 GMT
• Title: Self-Supervised Learning of Structured Dynamics from Videos
• Abstract summary: We study whether structured motion representations can be recovered from frozen features of a pretrained image vision transformer.<n>We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics.<n>We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics.
• Score: 76.23834498074805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding motion in video is a fundamental challenge for visual learning, as frame-to-frame change entangles two sources of dynamics: camera motion and object motion. This decomposition has remained underexplored in representation learning, partly because these factors are tightly coupled in natural videos and difficult to supervise separately. Yet recovering it is important for learning robust motion representations that separate meaningful object dynamics from camera-induced variation. We study whether such structured motion representations can be recovered from frozen features of a pretrained image vision transformer. We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics through future-feature prediction, rather than representing video change with a single entangled latent or with unstructured, spatially dense transition tokens. Training combines self-supervised learning on real video with weak supervision of scene dynamics on synthetic Kubric data. We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics. SDM outperforms backbone baselines using global CLS or average-pooled features, and compares favorably to strongly supervised representations such as VGGT on several probes, despite using substantially weaker supervision. These results suggest that pretrained image models can be readily repurposed into structured video-dynamics representations, providing a useful inductive bias for learning and analyzing latent video dynamics.

Related papers

• Analytic Dynamics: Learning Physics-Grounded Representation for Fast Intrinsic Dynamics Inference from Monocular Videos [2.5608298830470013]
  Inferring object dynamics from visual observations is essential for intelligent agents to reason about and interact with the physical world.<n>We propose Analytic Dynamics, a feed-forward dynamics inference framework that introduces an intermediate physics-grounded dynamics representation.
  arXiv  Detail & Related papers  (2026-08-31T16:08:04Z)
• Learning Explicit Physical Parameter Control and Benchmarking for Video Generation [21.331326205371628]
  We present PhyParam, a physics-guided image-to-video diffusion model that conditions on object-level forces, masses, friction, restitution, and scene-level gravity.<n>We also establish PhyParam-Bench, a benchmark for physical-law consistency in image-to-video generation.
  arXiv  Detail & Related papers  (2026-07-21T10:06:55Z)
• Local Motion Matters: A Deconstruct-Recompose Paradigm for Reinforcement Learning Pre-training from Videos [30.202149567220356]
  Existing methods typically treat the agent as an indivisible entity, modeling motion patterns globally.<n>We propose a novel Deconstruct-compose Paradigm for learning transferable local motion representations.<n>Our method effectively transfers to diverse robotic control and manipulation tasks, significantly improving sample efficiency and performance.
  arXiv  Detail & Related papers  (2026-07-01T11:36:27Z)
• PhyCo: Learning Controllable Physical Priors for Generative Motion [55.59209981836171]
  We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation.<n>Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback.
  arXiv  Detail & Related papers  (2026-04-30T17:53:03Z)
• From Static to Dynamic: Exploring Self-supervised Image-to-Video Representation Transfer Learning [79.76998842796515]
  Fine-tuning heavy modules may compromise inter-video semantic separability.<n>We propose the Consistency-Separability Trade-off Transfer Learning framework.<n>Experiments on eight image-pretrained models demonstrate consistent improvements across multiple levels of video tasks.
  arXiv  Detail & Related papers  (2026-03-27T16:56:50Z)
• Midway Network: Learning Representations for Recognition and Motion from Latent Dynamics [11.610274121239051]
  We present Midway Network, a new self-supervised learning architecture that is the first to learn strong visual representations for both object recognition and motion understanding solely from natural videos.<n>We show that Midway Network achieves strong performance on both semantic segmentation and optical flow tasks relative to prior self-supervised learning methods.
  arXiv  Detail & Related papers  (2025-10-07T04:07:44Z)
• Inferring Dynamic Physical Properties from Video Foundation Models [94.35979242947873]
  We study the task of predicting dynamic physical properties from videos.<n>We consider physical properties that require temporal information to be inferred: elasticity of a bouncing object, viscosity of a flowing liquid, and dynamic friction of an object sliding on a surface.
  arXiv  Detail & Related papers  (2025-10-02T17:59:50Z)
• SynMotion: Semantic-Visual Adaptation for Motion Customized Video Generation [56.90807453045657]
  SynMotion is a motion-customized video generation model that jointly leverages semantic guidance and visual adaptation.<n>At the semantic level, we introduce the dual-em semantic comprehension mechanism which disentangles subject and motion representations.<n>At the visual level, we integrate efficient motion adapters into a pre-trained video generation model to enhance motion fidelity and temporal coherence.
  arXiv  Detail & Related papers  (2025-06-30T10:09:32Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• Unfolding Videos Dynamics via Taylor Expansion [5.723852805622308]
  We present a new self-supervised dynamics learning strategy for videos: Video Time-Differentiation for Instance Discrimination (ViDiDi) ViDiDi observes different aspects of a video through various orders of temporal derivatives of its frame sequence. ViDiDi learns a single neural network that encodes a video and its temporal derivatives into consistent embeddings.
  arXiv  Detail & Related papers  (2024-09-04T01:41:09Z)
• Diffusion Priors for Dynamic View Synthesis from Monocular Videos [59.42406064983643]
  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. We first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. We distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields.
  arXiv  Detail & Related papers  (2024-01-10T23:26:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.