Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution

• URL: http://arxiv.org/abs/2605.25333v1
• Date: Mon, 25 May 2026 01:30:41 GMT
• Title: Teaching Video Generators to Remember: Eliciting Dynamic Memory for Out-of-Sight State Evolution
• Abstract summary: Video models should maintain evolving states when evidence is unobserved, yet current generators often freeze hidden states upon interruption.<n>We introduce ReMind, a framework eliciting dynamic memory behavior via memory-oriented data, event-aware training, and cacheated adaptation.
• Score: 25.63670341165374
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video world models should maintain evolving states when evidence is unobserved, yet current generators often freeze hidden states upon interruption. This is not simply a capacity problem: pretrained video diffusion transformers already possess KV-cache mechanisms capable of non-local retrieval, but they are rarely trained to use them as dynamic memory. We introduce ReMind, a framework eliciting dynamic memory behavior via memory-oriented data, event-aware training, and cache adaptation. Organized around a taxonomy of 100+ dynamic events, we build a camera-annotated training mixture combining VLM-filtered real videos, generated hard dynamics, synthetic camera loops, and memory-interruption augmentations. Each clip is converted into a frame graph with protected anchors, degraded intervals, and explicit temporal gaps. A node-structured curriculum, including node-drop, noisy memory, frontier continuation, and reference-cache training, forces the model to retrieve relevant past states across interruptions rather than relying solely on local continuity. PM-RoPE, an elegant camera-phase RoPE extension, unlocks spatiotemporal retrieval at a single-attention cost while preserving pretrained pathways. ReMind achieves the best overall scores on STEVO-Bench and recovery tasks. Furthermore, general image-to-video evaluations confirm this curriculum avoids catastrophic forgetting. We will open-source our code, data, and models.

Related papers

• Addressable Memory for Video World Models [50.40232922140905]
  We study visual persistence in interactive video world models.<n>We propose WorldTrace, a training-free memory framework for long-horizon visual persistence.
  arXiv  Detail & Related papers  (2026-08-07T16:55:57Z)
• TaoMate: Anchor-Guided Memory Bridging Evolving and Reference States for Real-Time Audio-Video Digital Human Generation [17.429355392513354]
  Real-time long-form digital-human generation relies on causal models to extend audio-visual content.<n>method is an anchor-guided persistent-memory framework for few-step joint audio-video generation.
  arXiv  Detail & Related papers  (2026-07-27T12:35:32Z)
• Online Neural Space Time Memory for Dynamic Novel View Synthesis [23.462185130690447]
  Test-Time Training (TTT) offers a powerful memory mechanism.<n>Standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes.<n>Our method demonstrates real-time, state-of-the-art performance on scenes with dynamic human motion as well as minute-scale online memorization.
  arXiv  Detail & Related papers  (2026-07-16T17:58:18Z)
• Echo-Infinity: Learning Evolving Memory for Real-Time Infinite Video Generation [59.01737077739414]
  We present Echo Infinity, an autoregressive (AR) framework towards real-time infinite video generation.<n>It employs a learnable evolving memory to dynamically filter, abstract, and compress any-length history at constant cost.<n>It achieves state-of-the-art performance, and, to our knowledge, demonstrates promising 24-hour (>1.3 M frames) real-time rollouts for the first time.
  arXiv  Detail & Related papers  (2026-06-03T07:09:01Z)
• Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation [89.70897512515477]
  Action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response.<n>Maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging.<n>Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from textitLatent--RGB Cycling, and the training--inference gap induced by the textiterror-free hypothesis.<n>We present textbfRobust Dreamer, a memory-augmented framework built around how to
  arXiv  Detail & Related papers  (2026-05-29T05:21:33Z)
• EM-Vid: Training-Free Entity-Centric Memory for Efficient and Consistent Multi-Shot Video Generation [18.052867824594347]
  Multi-shot video generation requires maintaining a consistent appearance of recurring entities across shots.<n>Recent autoregressive methods reuse previously generated frames as memory.<n>We propose an entity-centric memory in the form of an entity-indexed bank of latent patches.
  arXiv  Detail & Related papers  (2026-05-22T13:20:29Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• Echo-Forcing: A Scene Memory Framework for Interactive Long Video Generation [48.476317015122625]
  Echo-Forcing is a training-free scene memory framework for interactive long video generation.<n>It supports smooth transitions, hard cuts, and long-range scene recall under a bounded cache budget.
  arXiv  Detail & Related papers  (2026-05-15T14:33:09Z)
• SWIFT: Prompt-Adaptive Memory for Efficient Interactive Long Video Generation [31.105007908298003]
  Streaming long-video generation faces a central challenge in continuous semantic switching.<n>Current approaches rely on cache rebuilding at prompt boundaries or fixed memory budgets.<n>We present SWIFT, a training-free framework for multi-prompt long-video generation.
  arXiv  Detail & Related papers  (2026-05-10T09:37:56Z)
• Out of Sight but Not Out of Mind: Hybrid Memory for Dynamic Video World Models [56.44348799741838]
  We introduce Hybrid Memory, a novel paradigm requiring models to simultaneously act as precise archivists for static backgrounds and vigilant trackers for dynamic subjects.<n>We construct HM-World, the first large-scale video dataset dedicated to hybrid memory.<n>We also propose HyDRA, a specialized memory architecture that compresses memory into tokens and utilizes atemporal relevance-driven retrieval mechanism.
  arXiv  Detail & Related papers  (2026-03-26T17:56:01Z)
• Recurrent Dynamic Embedding for Video Object Segmentation [54.52527157232795]
  We propose a Recurrent Dynamic Embedding (RDE) to build a memory bank of constant size. We propose an unbiased guidance loss during the training stage, which makes SAM more robust in long videos. We also design a novel self-correction strategy so that the network can repair the embeddings of masks with different qualities in the memory bank.
  arXiv  Detail & Related papers  (2022-05-08T02:24:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.