Daily Paper Briefing

Abstract Overview

PhysStream is an autoregressive image-to-video generation method for physics-grounded, interactive control in multi-object tabletop scenes. The model takes a single input image and lets users inject sparse per-object velocity-increment signals at chosen timesteps, rather than requiring a full motion plan in advance. To maintain physical and geometric consistency during streaming generation, it uses structured scene memory composed of positional maps from monocular depth estimation and object-tracking maps derived online from previously generated frames. The training procedure is split into two stages: first adapting a bidirectional video model to motion control, then converting it to a causal autoregressive model that incorporates scene memory. Experiments cover synthetic benchmarks, in-the-wild scenes, real-world captures, long-horizon generation, and finetuned non-rigid settings.

Novelty

The paper's main novelty is combining direct scene-level interactive control with physics-grounded velocity inputs in an end-to-end autoregressive video generator. It further introduces structured scene memory—online positional and tracking maps—as a history-dependent conditioning mechanism, together with a two-stage training strategy that avoids information leakage when moving from bidirectional to causal generation.

Results

On synthetic multi-object interactive benchmarks, PhysStream achieves the best physics-sensitive scores among compared methods, including an FVMD of 787.0 and Traj-ADE of 40.24, reducing motion distribution distance by 33% and trajectory error by 12% versus the strongest baselines. On in-the-wild comparisons, it receives the highest reported MLLM scores and is preferred by human evaluators in 91.8% of physical plausibility comparisons, 88.4% for motion accuracy, and 91.2% for visual quality. The method also maintains high control responsiveness over 301-frame long-horizon sequences, though consistency gradually declines with accumulated appearance drift.

Key Points

1. PhysStream enables streaming image-to-video generation with sparse, per-object velocity-increment control that can be applied interactively during generation.
2. The method augments autoregressive generation with structured scene memory from online depth-based positional maps and SAM2-based object-tracking maps, trained via a two-stage bidirectional-to-causal pipeline.
3. Empirically, it improves motion-control adherence and physical plausibility over prior controllable baselines on synthetic and in-the-wild evaluations, while also showing long-horizon control capability.

References

• arXiv: https://arxiv.org/abs/2609.17521v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2609.17521v1
• Hugging Face Papers: https://huggingface.co/papers/2609.17521
• Project: https://czzzzh.github.io/PhysStream