Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models

• URL: http://arxiv.org/abs/2603.25716v2
• Date: Sat, 28 Mar 2026 08:29:52 GMT
• Title: Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models
• Abstract summary: We introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects.<n>We construct HM-World, the first large-scale video dataset dedicated to hybrid memory.<n>We also propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes atemporal relevance-driven retrieval mechanism.
• Score: 56.44348799741838
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video world models have shown immense potential in simulating the physical world, yet existing memory mechanisms primarily treat environments as static canvases. When dynamic subjects hide out of sight and later re-emerge, current methods often struggle, leading to frozen, distorted, or vanishing subjects. To address this, we introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects, ensuring motion continuity during out-of-view intervals. To facilitate research in this direction, we construct HM-World, the first large-scale video dataset dedicated to hybrid memory. It features 59K high-fidelity clips with decoupled camera and subject trajectories, encompassing 17 diverse scenes, 49 distinct subjects, and meticulously designed exit-entry events to rigorously evaluate hybrid coherence. Furthermore, we propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes a spatiotemporal relevance-driven retrieval mechanism. By selectively attending to relevant motion cues, HyDRA effectively preserves the identity and motion of hidden subjects. Extensive experiments on HM-World demonstrate that our method significantly outperforms state-of-the-art approaches in both dynamic subject consistency and overall generation quality. Code is publicly available at https://github.com/H-EmbodVis/HyDRA.

Related papers

• Wonder: Video World Model Done Better [70.28870858365214]
  We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration.<n>Given an image or a conditional video, Wonder constructs a playable world where users can navigate interactively by moving the camera.<n>We introduce a novel camera conditioning with a dense coordinate field whose renderings provide spatially aligned motion and orientation cues.
  arXiv  Detail & Related papers  (2026-07-28T17:45:25Z)
• MemoryWAM: Efficient World Action Modeling with Persistent Memory [80.90899269062128]
  We introduce MemoryWAM, a world action model with efficient persistent memory.<n>A tailored attention mechanism enables retrieval of both detailed short-term context and compressed long-term context.<n>MemoryWAM outperforms strong vision-language-action (VLA) and WAM baselines in both simulation and the real world.
  arXiv  Detail & Related papers  (2026-06-18T17:59:51Z)
• Mem-World: Memory-Augmented Action-Conditioned World Models for Persistent Robot Manipulation [55.42006264038458]
  Action-conditioned world models have emerged as a promising paradigm for robot learning.<n>Mem-World is a memory-augmented multi-view action-conditioned world model.<n>We present W-VMem, a 4D wrist-view-centered surfel-indexed memory that anchors historical observations to temporally evolving surface elements.
  arXiv  Detail & Related papers  (2026-06-17T11:42:00Z)
• Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution [25.63670341165374]
  Video models should maintain evolving states when evidence is unobserved, yet current generators often freeze hidden states upon interruption.<n>We introduce ReMind, a framework eliciting dynamic memory behavior via memory-oriented data, event-aware training, and cacheated adaptation.
  arXiv  Detail & Related papers  (2026-05-25T01:30:41Z)
• AGILE: Hand-Object Interaction Reconstruction from Video via Agentic Generation [45.753757870577196]
  We introduce AGILE, a robust framework that shifts the paradigm from reconstruction to agentic generation for interaction learning.<n>We show that AGILE outperforms baselines in global geometric accuracy while demonstrating exceptional robustness on challenging sequences where prior art frequently collapses.
  arXiv  Detail & Related papers  (2026-02-04T15:42:58Z)
• TeleWorld: Towards Dynamic Multimodal Synthesis with a 4D World Model [53.555353366322464]
  We present TeleWorld, a real-time multimodal 4D world modeling framework that unifies video generation, dynamic scene reconstruction, and long-term world memory within a closed-loop system.<n>Our approach achieves seamless integration of dynamic object modeling and static scene representation within a unified 4D framework, advancing world models toward practical, interactive, and computationally accessible synthesis systems.
  arXiv  Detail & Related papers  (2025-12-31T18:31:46Z)
• Prismatic World Model: Learning Compositional Dynamics for Planning in Hybrid Systems [38.4555621948915]
  Prismatic World Model (PRISM-WM) is designed to decompose complex hybrid dynamics into composable primitives.<n>PRISM-WM significantly reduces rollout drift by accurately modeling sharp mode transitions in system dynamics.
  arXiv  Detail & Related papers  (2025-12-09T09:40:34Z)
• VideoSSM: Autoregressive Long Video Generation with Hybrid State-Space Memory [42.2374676860638]
  Autoregressive (AR) diffusion enables streaming, interactive long-video generation by producing frames causally.<n>Maintaining coherence over minute-scale horizons remains challenging due to accumulated errors, motion drift, and content repetition.<n>We propose VideoSSM, a Long Video Model that unifies AR diffusion with a hybrid state-space memory.
  arXiv  Detail & Related papers  (2025-12-04T07:06:02Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• Mem4D: Decoupling Static and Dynamic Memory for Dynamic Scene Reconstruction [17.587320705104343]
  We propose a novel framework that decouples the modeling of static geometry and dynamic motion.<n>Mem4D simultaneously maintains static geometry with global consistency and reconstructs dynamic elements with high fidelity.
  arXiv  Detail & Related papers  (2025-08-11T12:23:31Z)
• Video World Models with Long-term Spatial Memory [110.530715838396]
  We introduce a novel framework to enhance long-term consistency of video world models.<n>Our framework includes mechanisms to store and retrieve information from the long-term spatial memory.<n>Our evaluations show improved quality, consistency, and context length compared to relevant baselines.
  arXiv  Detail & Related papers  (2025-06-05T17:42:34Z)
• EgoGaussian: Dynamic Scene Understanding from Egocentric Video with 3D Gaussian Splatting [95.44545809256473]
  EgoGaussian is a method capable of simultaneously reconstructing 3D scenes and dynamically tracking 3D object motion from RGB egocentric input alone. We show significant improvements in terms of both dynamic object and background reconstruction quality compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2024-06-28T10:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.