Latent Spatial Memory for Video World Models

• URL: http://arxiv.org/abs/2606.09828v1
• Date: Mon, 08 Jun 2026 17:59:54 GMT
• Title: Latent Spatial Memory for Video World Models
• Abstract summary: We introduce emphlatent spatial memory for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction.<n>Experiments show that latent spatial memory achieves up to textbf10.57$times$ faster end-to-end video generation and textbf55$times$ reduction in memory footprint relative to explicit 3D baselines.
• Score: 45.02259712072663
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video world models that maintain 3D spatial consistency across generated frames typically rely on explicit point cloud memory constructed in RGB space. This design is both computationally expensive, requiring repeated rendering and VAE encoding, and inherently lossy, as the round trip through pixel space discards rich features of the learned latent representation. In this paper, we introduce \emph{latent spatial memory} for video world models, a persistent 3D cache that stores scene information directly in the diffusion latent space, avoiding pixel-space reconstruction. Building on this, we propose Mirage, a latent-space spatial memory framework that constructs the memory by lifting latent tokens into 3D via depth-guided back-projection and queries it by synthesizing novel views through direct latent-space warping. This unified formulation eliminates both the information loss of pixel-space reconstruction and the computational burden of repeated encoding and rendering. Experiments show that latent spatial memory achieves up to \textbf{10.57}$\times$ faster end-to-end video generation and \textbf{55}$\times$ reduction in memory footprint relative to explicit 3D baselines. Leveraging the geometric prior of the diffusion model, Mirage attains state-of-the-art performance on WorldScore and strong reconstruction quality on RealEstate10K.

Related papers

• Sparse auto-regressive modeling for scene generation from multi-view images [20.272184280044083]
  We introduce SPAR3S, a sparse voxel-aligned 3D latent generative model for conditional scene completion.<n>Our key insight is to formulate 3D scene generation in a structured, compact, voxel-aligned 3D latent space.<n>We demonstrate the effectiveness of our method on synthetic indoor scenes, achieving higher novel-view quality than prior work.
  arXiv  Detail & Related papers  (2026-09-03T14:41:34Z)
• Geometry-Aware Implicit Memory for Video World Models [29.522291951915577]
  GIM-World is a geometry-aware implicit memory framework for video world models.<n>A camera-queryable geometry head distills 3D scene structure from a frozen foundation model into the memory during training.<n>Experiments on MIND show that GIM-World better preserves long-horizon geometric and visual consistency than both explicit- and implicit-memory baselines.
  arXiv  Detail & Related papers  (2026-06-01T16:08:43Z)
• Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation [89.70897512515477]
  Action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response.<n>Maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging.<n>Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from textitLatent--RGB Cycling, and the training--inference gap induced by the textiterror-free hypothesis.<n>We present textbfRobust Dreamer, a memory-augmented framework built around how to
  arXiv  Detail & Related papers  (2026-05-29T05:21:33Z)
• SpaceMind++: Toward Allocentric Cognitive Maps for Spatially Grounded Video MLLMs [6.871639227720094]
  We propose SpaceMind, a video MLLM architecture that builds a voxelized cognitive map from RGB videos.<n>This map reorganizes fragmented egocentric observations into a shared 3D metric representation.<n>We introduce Coordinate Deep Iterative Fusion, a new mechanism that relays map-level spatial knowledge back into the original 2D visual features.
  arXiv  Detail & Related papers  (2026-05-10T10:01:57Z)
• I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation [56.33710337846449]
  I3DM is a novel implicit 3D-aware memory mechanism for consistent video scene generation.<n>At the core of our approach is a 3D-aware memory retrieval strategy.<n>To fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module.
  arXiv  Detail & Related papers  (2026-03-24T16:45:40Z)
• Spatia: Video Generation with Updatable Spatial Memory [60.21619361473996]
  Spatia is a spatial memory-aware video generation framework that preserves a 3D scene point cloud as persistent spatial memory.<n>Spatia iteratively generates video clips conditioned on this spatial memory and continuously updates it through visual SLAM.<n>Spatia enables applications such as explicit camera control and 3D-aware interactive editing, providing a geometrically grounded framework for scalable, memory-driven video generation.
  arXiv  Detail & Related papers  (2025-12-17T18:59:59Z)
• OnlineSplatter: Pose-Free Online 3D Reconstruction for Free-Moving Objects [58.38338242973447]
  OnlineSplatter is a novel framework generating high-quality, object-centric 3D Gaussians directly from RGB frames.<n>Our approach anchors reconstruction using the first frame and progressively refines the object representation through a dense Gaussian primitive field.<n>Our core contribution is a dual-key memory module combining latent appearance-geometry keys with explicit directional keys.
  arXiv  Detail & Related papers  (2025-10-23T14:37:25Z)
• Memory Forcing: Spatio-Temporal Memory for Consistent Scene Generation on Minecraft [45.363427511806385]
  Memory Forcing is a learning framework that pairs training protocols with a geometry-indexed spatial memory.<n>We show that Memory Forcing achieves superior long-term spatial consistency and generative quality across diverse environments.
  arXiv  Detail & Related papers  (2025-10-03T17:35:16Z)
• Point3R: Streaming 3D Reconstruction with Explicit Spatial Pointer Memory [72.75478398447396]
  We propose Point3R, an online framework targeting dense streaming 3D reconstruction.<n>To be specific, we maintain an explicit spatial pointer memory directly associated with the 3D structure of the current scene.<n>Our method achieves competitive or state-of-the-art performance on various tasks with low training costs.
  arXiv  Detail & Related papers  (2025-07-03T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.