Tempered Self-Similarity Alignment for Physically Plausible Video Generation

• URL: http://arxiv.org/abs/2605.24962v1
• Date: Sun, 24 May 2026 09:28:05 GMT
• Title: Tempered Self-Similarity Alignment for Physically Plausible Video Generation
• Abstract summary: Self-similarity loss transforms into probabilistic correspondence and trains the video generative model to align its correspondence with those of the visual foundation model on dynamically changing regions.<n>Our method demonstrates substantial improvements in physical plausibility across diverse interaction scenarios, validating the effectiveness of transferring relational knowledge for video generation.
• Score: 83.40337664171939
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite remarkable advances in video generative models, they still struggle to generate physically realistic videos, frequently exhibiting appearance drift, implausible motion, and temporal inconsistencies. In this work, we address this limitation by transferring relational knowledge encoded in spatio-temporal self-similarity (STSS) from visual foundation models into video generative models. STSS represents pairwise similarities among features across space and time, revealing the relational structure of how objects interact with other entities throughout a video, effectively capturing real-world dynamics, including object motion and semantic transformations. To transfer this relational knowledge, we propose Tempered Self-similarity Alignment (TSA) loss, which transforms STSS into probabilistic correspondence distributions and trains the video generative model to align its correspondence distributions with those of the visual foundation model on dynamically changing regions. Evaluated on VideoPhy and VideoPhy2 benchmarks, our method demonstrates substantial improvements in physical plausibility across diverse interaction scenarios, validating the effectiveness of transferring relational knowledge for physically realistic video generation.

Related papers

• Self-Supervised Learning of Structured Dynamics from Videos [76.23834498074805]
  We study whether structured motion representations can be recovered from frozen features of a pretrained image vision transformer.<n>We propose the Structured Dynamics Model (SDM), which explicitly separates the dominant source of temporal change from residual dynamics.<n>We evaluate SDM on ProbeMotion, a new evaluation suite spanning synthetic and real videos with camera motion, object motion, and combined dynamics.
  arXiv  Detail & Related papers  (2026-07-23T17:55:07Z)
• CausalMotion: Structured Physical Reasoning as Keyframe and Trajectory Guidance for Training-Free Video Generation [31.482087672315895]
  textbfCausalMotion injects explicit physical reasoning into video generation through structured intermediate representations.<n>Our method consistently improves physical plausibility and temporal coherence, particularly in dynamics-intensive scenarios.
  arXiv  Detail & Related papers  (2026-06-12T09:57:01Z)
• PhyCo: Learning Controllable Physical Priors for Generative Motion [55.59209981836171]
  We present PhyCo, a framework that introduces continuous, interpretable, and physically grounded control into video generation.<n>Our approach integrates three key components: (i) a large-scale dataset of over 100K photorealistic simulation videos where friction, restitution, deformation, and force are systematically varied across diverse scenarios; (ii) physics-supervised fine-tuning of a pretrained diffusion model using a ControlNet conditioned on pixel-aligned physical property maps; and (iii) VLM-guided reward optimization, where a fine-tuned vision-language model evaluates generated videos with targeted physics queries and provides differentiable feedback.
  arXiv  Detail & Related papers  (2026-04-30T17:53:03Z)
• Learning Transferable Temporal Primitives for Video Reasoning via Synthetic Videos [52.00944453189226]
  We introduce SynRL, a post-training framework that teaches models temporal primitives.<n>We decompose temporal understanding into short-term primitives (speed, direction) and long-term cognitive primitives.<n>Despite training on simple geometric shapes, SynRL achieves substantial improvements across 15 benchmarks spanning temporal grounding, complex reasoning, and general video understanding.
  arXiv  Detail & Related papers  (2026-03-18T13:10:47Z)
• AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation [45.753757870577196]
  We introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning.<n>We show that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses.
  arXiv  Detail & Related papers  (2026-02-04T15:42:58Z)
• BindWeave: Subject-Consistent Video Generation via Cross-Modal Integration [56.98981194478512]
  We propose a unified framework that handles a broad range of subject-to-video scenarios.<n>We introduce an MLLM-DiT framework in which a pretrained multimodal large language model performs deep cross-modal reasoning to ground entities.<n>Experiments on the OpenS2V benchmark demonstrate that our method achieves superior performance across subject consistency, naturalness, and text relevance in generated videos.
  arXiv  Detail & Related papers  (2025-10-01T02:41:11Z)
• Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation [17.301302433153467]
  We investigate the effects of existing fine-tuning video generation approaches on structured driving datasets.<n>We show that although visual fidelity improves, spatial accuracy in modeling dynamic elements may degrade.
  arXiv  Detail & Related papers  (2025-08-22T16:35:19Z)
• InterDyn: Controllable Interactive Dynamics with Video Diffusion Models [50.38647583839384]
  We propose InterDyn, a framework that generates videos of interactive dynamics given an initial frame and a control signal encoding the motion of a driving object or actor.<n>Our key insight is that large video generation models can act as both neurals and implicit physics simulators'', having learned interactive dynamics from large-scale video data.
  arXiv  Detail & Related papers  (2024-12-16T13:57:02Z)
• Relational Self-Attention: What's Missing in Attention for Video Understanding [52.38780998425556]
  We introduce a relational feature transform, dubbed the relational self-attention (RSA) Our experiments and ablation studies show that the RSA network substantially outperforms convolution and self-attention counterparts.
  arXiv  Detail & Related papers  (2021-11-02T15:36:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.