Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation

• URL: http://arxiv.org/abs/2605.30855v2
• Date: Mon, 01 Jun 2026 09:19:36 GMT
• Title: Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation
• Abstract summary: Action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response.<n>Maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging.<n>Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from textitLatent--RGB Cycling, and the training--inference gap induced by the textiterror-free hypothesis.<n>We present textbfRobust Dreamer, a memory-augmented framework built around how to
• Score: 89.70897512515477
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Frame-wise action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response. However, maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging. Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from \textit{Latent--RGB Cycling}, where generated latents are repeatedly decoded to RGB and re-encoded for future conditioning, and the training--inference gap induced by the \textit{error-free hypothesis}, where clean training memory fails to match prediction-corrupted inference memory. To address these challenges, we present \textbf{Robust Dreamer}, a memory-augmented framework built around how to design 3D memory and how to use it robustly. First, we introduce \textbf{Latent Gaussian Memory}, which anchors diffusion latents inherited from the generation process to Gaussian primitives and recalls them via latent-space Gaussian splatting. This provides dense, geometry-aware, view-aligned conditioning while avoiding accumulated degradation from repeated VAE conversion. Second, we propose \textbf{Deviation Learning with Dynamic Deviation Archive}, which synthesizes rollout-induced latent deviations through a one-step approximation, stores them by autoregressive stage and denoising timestamp, and injects them into historical memory during training. This exposes the generator to realistic corrupted memory states and teaches internal correction before inference. Experiments on ScanNet, DL3DV, and OmniWorldGame demonstrate state-of-the-art long-horizon performance.

Related papers

• AlayaWorld: Interactive Long-Horizon World Modeling - Full Technical Report (v1.1) [39.20781986625199]
  We revise how conditioning signals are represented and integrated into the model.<n>New design is guided by a simple principle: conditioning signals should match the generated content as closely as possible in both latent representation and temporal structure.
  arXiv  Detail & Related papers  (2026-08-13T17:21:03Z)
• Self Gradient Forcing: Native Long Video Extrapolation [60.00513917489133]
  Self Gradient Forcing restores historical context-gradient gap.<n>Code and models will be released to advance research on autoregressive video generation.
  arXiv  Detail & Related papers  (2026-07-22T16:59:23Z)
• SSD: Spatially Speculative Decoding Accelerates Autoregressive Image Generation [19.15882131245149]
  We introduce Spatially Speculative Decoding, a framework that aligns the predictive objective with the natural geometry of images.<n>Our results suggest that respecting the underlying geometry of vision unlocks massive computational efficiencies.
  arXiv  Detail & Related papers  (2026-06-18T17:52:30Z)
• GaussianDream: A Feed-Forward 3D Gaussian World Model for Robotic Manipulation [54.671815855499034]
  Vision-language-action (VLA) policies have advanced language-conditioned robotic manipulation by transferring semantic priors to action generation.<n>Standard action-imitation learning often lacks sufficient modeling of explicit 3D spatial information, dense geometric supervision, and future environment evolution.<n>We propose textbfGaussianDream, a feed-forward 3D Gaussian world-model plug-in.
  arXiv  Detail & Related papers  (2026-05-20T05:51:30Z)
• Scal3R: Scalable Test-Time Training for Large-Scale 3D Reconstruction [50.5449251266956]
  This paper addresses the task of large-scale 3D scene reconstruction from long video sequences.<n>Recent feed-forward reconstruction models have shown promising results by directly regressing 3D geometry from RGB images without explicit 3D priors or geometric constraints.<n>We propose a novel neural global context representation that efficiently compresses and retains long-range scene information.
  arXiv  Detail & Related papers  (2026-04-09T17:59:50Z)
• I3DM: Implicit 3D-aware Memory Retrieval and Injection for Consistent Video Scene Generation [56.33710337846449]
  I3DM is a novel implicit 3D-aware memory mechanism for consistent video scene generation.<n>At the core of our approach is a 3D-aware memory retrieval strategy.<n>To fully utilize the retrieved historical frames, we introduce a 3D-aligned memory injection module.
  arXiv  Detail & Related papers  (2026-03-24T16:45:40Z)
• RELIC: Interactive Video World Model with Long-Horizon Memory [74.81433479334821]
  A truly interactive world model requires real-time long-horizon streaming, consistent spatial memory, and precise user control.<n>We present RELIC, a unified framework that tackles these three challenges altogether.<n>Given a single image and a text description, RELIC enables memory-aware, long-duration exploration of arbitrary scenes in real time.
  arXiv  Detail & Related papers  (2025-12-03T18:29:20Z)
• Stable Video Infinity: Infinite-Length Video Generation with Error Recycling [76.91310169118408]
  We propose Stable Video Infinity (SVI) that is able to generate infinite-length videos with high temporal consistency, plausible scene transitions, and controllable streaming storylines.<n> SVI incorporates Error-Recycling Fine-Tuning, a new type of efficient training that recycles the Diffusion Transformer's self-generated errors into supervisory prompts.<n>We evaluate SVI on three benchmarks, including consistent, creative, and conditional settings, thoroughly verifying its versatility and state-of-the-art role.
  arXiv  Detail & Related papers  (2025-10-10T09:45:46Z)
• Forge4D: Feed-Forward 4D Human Reconstruction and Interpolation from Uncalibrated Sparse-view Videos [27.595035122927204]
  We present a feed-forward 4D human reconstruction and model that efficientlycalibrates temporally aligned representations from uncalibrated sparse-view videos.<n>For novel time, we design a novel motion prediction module to predict dense motions for each 3D Gaussian between two adjacent frames.<n>Experiments demonstrate the effectiveness of our model on both in-domain and out-of-domain datasets.
  arXiv  Detail & Related papers  (2025-09-29T02:47:14Z)
• JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation [22.956416709470503]
  Vision-and-Language Navigation requires an embodied agent to navigate through unseen environments, guided by natural language instructions and a continuous video stream.<n>Recent advances in VLN have been driven by the powerful semantic understanding of Multimodal Large Language Models.<n>We propose JanusVLN, a novel VLN framework featuring a dual implicit neural memory that models spatial-geometric and visual-semantic memory as separate, compact, and fixed-size neural representations.
  arXiv  Detail & Related papers  (2025-09-26T16:29:37Z)
• VoxelSplat: Dynamic Gaussian Splatting as an Effective Loss for Occupancy and Flow Prediction [46.31516096522758]
  Recent advancements in camera-based occupancy prediction have focused on the simultaneous prediction of 3D semantics and scene flow.<n>We propose a novel regularization framework called VoxelSplat to address these challenges and their underlying causes.<n>Our framework uses the predicted scene flow to model the motion of Gaussians, and is thus able to learn the scene flow of moving objects in a self-supervised manner.
  arXiv  Detail & Related papers  (2025-06-05T20:19:35Z)
• StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D [88.66678730537777]
  We present StableDreamer, a methodology incorporating three advances. First, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. Second, our analysis shows that while image-space diffusion contributes to geometric precision, latent-space diffusion is crucial for vivid color rendition.
  arXiv  Detail & Related papers  (2023-12-02T02:27:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.