ReWorld: An Interactive World Model with Long-Horizon Memory
Abstract Overview
ReWorld is an interactive streaming world model designed to simultaneously follow user camera actions, preserve memory of previously visited places, and run in real time. The core design separates control and memory during training: most attention heads learn short-window reactive control, while a smaller set learns long-range retrieval, with random head routing preventing either function from binding to fixed heads. At inference, the full history is compressed into a bounded KV cache supported by a pose-indexed landmark bank that stores and retrieves spatially relevant past chunks under a fixed memory budget. The system is trained on a metric-scale-aligned eight-source dataset spanning Unreal-rendered scenes, game footage, and real-world video, and uses a LoRA-based four-step distillation path so one backbone can serve both high-fidelity and real-time modes.
Novelty
The paper’s main novelty is the explicit decoupling of control and long-horizon memory by training different attention windows within the same model and randomizing which heads are global versus local. It also combines this with a bounded-memory inference scheme based on landmark consolidation and pose-based retrieval, plus chunk-drop training to match the sparse cache seen at deployment.
Results
Across a shared controllability benchmark against six recent baselines, ReWorld achieves the best overall rotation error (11.95°) and best camera-motion consistency, while also obtaining the best mean VBench quality score (0.850). On long-horizon palindrome memory tests, it maintains strong revisit similarity at k=96 latents (SSIM 0.384, LPIPS 0.332, DINO 0.932, ORB 0.379) while traversing a longer path than most baselines, and ablations show that the landmark-bank cache improves far-horizon recall under a fixed 12-chunk budget.
Key Points
- ReWorld separates action control and spatial memory during training using mixed per-head attention windows and random head routing.
- Its bounded-memory inference uses a pose-indexed landmark bank with chunk consolidation and retrieval, supported by chunk-drop training to reduce train-test mismatch.
- The method shows strong empirical performance in controllability, long-horizon revisit memory, and overall video quality while supporting four-step real-time inference.
References
- arXiv: https://arxiv.org/abs/2608.23565v1
- Fugu-MT: https://fugumt.com/fugumt/paper_check/2608.23565v1
- Hugging Face Papers: https://huggingface.co/papers/2608.23565
- Project: https://zhifeichen097.github.io/ReWorld/