PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control

• URL: http://arxiv.org/abs/2609.17521v1
• Date: Tue, 15 Sep 2026 17:55:13 GMT
• Title: PhysStream: Streaming Physics-Grounded Video Generation with Structured Scene Memory and Fine-Grained Motion Control
• Abstract summary: PhysStream is an autoregressive model for physics-grounded image-to-video synthesis.<n>It supports fine-grained motion control via sparse velocity-increment signals.<n>PhysStream is preferred by human evaluators in over 85% of in-the-wild comparisons.
• Score: 85.54736105647055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Interactive control for video generation is moving from coarse prompts toward fine-grained, physically meaningful manipulation of dynamic scenes. Yet existing controllable methods either require the full control schedule before generation starts, or use pixel-space signals that dictate object positions rather than physical dynamics. To address these limitations, we propose PhysStream, an autoregressive model for physics-grounded image-to-video synthesis that incorporates structured scene memory---positional maps and object tracking maps derived online from previously generated frames---and supports fine-grained motion control via sparse velocity-increment signals that encode physical quantities, letting the model learn the underlying dynamics. We train our model in two stages: a bidirectional model is first finetuned with motion-control conditioning, then a causal autoregressive model is trained with additional structured scene memory, further improving physical consistency. PhysStream enables interactive, mid-generation control over multi-object tabletop rigid-body scenes---a capability not supported by prior methods---reducing motion distribution distance (FVMD) by 33% and trajectory error by 12% over the strongest baselines on synthetic benchmarks, and is preferred by human evaluators in over 85% of in-the-wild comparisons. Please check our website for more details: https://czzzzh.github.io/PhysStream

Related papers

• Physics-Grounded Fluid Video Generation with a Simulation Dataset and Dual-Stream Optical-Flow Supervision [78.04385989413382]
  Video diffusion models generate visually compelling content but routinely violate elementary physics when the subject involves fluids.<n>We build a physics-simulation fluid dataset combining 1,638 MPM-simulated pouring/sloshing videos with 2,320 keyword-filtered real pouring videos.<n>We introduce a dual-stream image-to-video architecture built on a pretrained diffusion-filtereder video generator.
  arXiv  Detail & Related papers  (2026-07-28T06:13:18Z)
• Learning Explicit Physical Parameter Control and Benchmarking for Video Generation [21.331326205371628]
  We present PhyParam, a physics-guided image-to-video diffusion model that conditions on object-level forces, masses, friction, restitution, and scene-level gravity.<n>We also establish PhyParam-Bench, a benchmark for physical-law consistency in image-to-video generation.
  arXiv  Detail & Related papers  (2026-07-21T10:06:55Z)
• TelePhysics: Physics-Grounded Multi-Object Scene Generation from a Single Image with Real-Time Interaction [51.01447538245441]
  We propose TelePhysics, a training-free framework that converts a single image into a physically consistent and controllable video.<n>By representing the full scene geometry in a unified spatial coordinate system, TelePhysics resolves object penetration and alignment ambiguity.<n> Experimental results demonstrate that TelePhysics substantially outperforms prior methods in physical fidelity, spatial coherence, and controllability.
  arXiv  Detail & Related papers  (2026-05-19T08:16:44Z)
• DynaVid: Learning to Generate Highly Dynamic Videos using Synthetic Motion Data [51.316274891736164]
  DynaVid is a video synthesis framework that leverages synthetic motion data in training.<n>We show that DynaVid improves the realism and controllability in dynamic motion generation and camera motion control.
  arXiv  Detail & Related papers  (2026-04-02T06:12:38Z)
• Physical Simulator In-the-Loop Video Generation [96.87054314612142]
  Physical Simulator In-the-loop Video Generation (PSIVG) is a novel framework that integrates a physical simulator into the video diffusion process.<n>PSIVG produces videos that better adhere to real-world physics while preserving visual quality and diversity.
  arXiv  Detail & Related papers  (2026-03-06T15:48:25Z)
• PhysCtrl: Generative Physics for Controllable and Physics-Grounded Video Generation [53.06495362038348]
  Existing generation models excel at producing photo-realistic videos from text or images, but often lack physical plausibility and 3D controllability.<n>We introduce PhysCtrl, a novel framework for physics-grounded image-to-video generation with physical parameters and force control.<n> Experiments show that PhysCtrl generates realistic, physics-grounded motion trajectories which, when used to drive image-to-video models, yield high-fidelity, controllable videos.
  arXiv  Detail & Related papers  (2025-09-24T17:58:04Z)
• Optimal-state Dynamics Estimation for Physics-based Human Motion Capture from Videos [6.093379844890164]
  We propose a novel method to selectively incorporate the physics models with the kinematics observations in an online setting.<n>A recurrent neural network is introduced to realize a Kalman filter that attentively balances the kinematics input and simulated motion.<n>The proposed approach excels in the physics-based human pose estimation task and demonstrates the physical plausibility of the predictive dynamics.
  arXiv  Detail & Related papers  (2024-10-10T10:24:59Z)
• Physics-based Scene Layout Generation from Human Motion [21.939444709132395]
  We present a physics-based approach that simultaneously optimize a scene layout generator and simulates a moving human in a physics simulator. We use reinforcement learning to perform a dual-optimization of both the character motion imitation controller and the scene layout generator. We evaluate our method using motions from SAMP and PROX, and demonstrate physically plausible scene layout reconstruction compared with the previous kinematics-based method.
  arXiv  Detail & Related papers  (2024-05-21T02:36:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.