The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL

• URL: http://arxiv.org/abs/2607.19749v1
• Date: Wed, 22 Jul 2026 04:46:49 GMT
• Title: The World Model Remembers, the Actor Forgets: Dream Rehearsal for Continual Model-Based RL
• Abstract summary: Reinforcement learning agents forget catastrophically when trained on task sequences.<n>We show that forgetting in this regime is a channel problem, not a memory problem.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model-based reinforcement-learning agents of the DreamerV3 family forget catastrophically when trained on task sequences, even when an unbounded replay buffer preserves every earlier experience. We ask a question the continual-RL literature has assumed an answer to but never measured: which component forgets? Under never-clear replay, pre-registered component-level probes (n=3 seeds throughout) show that the world model retains essentially everything measurable about old tasks -- reward discrimination (retention ratio ~1.0), value estimates, and termination structure -- while the actor's behavior collapses. Forgetting in this regime is a channel problem, not a memory problem. We demonstrate this by intervention: with the world model frozen and identical imagined rollouts, reinforcement learning in imagination fails to recover a lost skill (0/3 seeds), while supervised self-imitation on the world model's own graded dreams recovers it on 3/3 seeds with zero environment interaction. Interleaved during training, this graded dream rehearsal yields a task-label-free, parameter-constant continual learner: 3/3 four-task chains retained where plain replay passes 0/3, 3/3 eight-task chains, and consistent gains over matched real-episode cloning (paired difference +0.13, bootstrap 95% CI [0.07, 0.24], complete seed separation). The dream-grading step is load-bearing: we characterize two scoring failure modes, provide an offline selection gauge that caught both before they contaminated results, and give a realized-first grading rule that closes them. All experiments were pre-registered with committed protocols; every refuted hypothesis is reported.

Related papers

• The Intervention Gap in Latent World Models [0.0]
  Planning-time intervention fidelity is a distinct, measurable property of a learned world model.<n>Across released TD-MPC2 checkpoint sizes, episode return falls as an operator-error diagnostic on task observables grows.<n>A self-supervised world model trained without task signal preserves the same operator substantially better than a task-anchored model on the shared task.
  arXiv  Detail & Related papers  (2026-08-30T19:47:51Z)
• Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem [16.147107064302435]
  We develop a protocol and training corpus that captures forget-retain structure for real-world factual domains.<n>We introduce JensUn++, an unlearning algorithm that achieves the best forget-retain utility trade-off across three LLMs.
  arXiv  Detail & Related papers  (2026-07-10T09:31:29Z)
• The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory [0.0]
  The effect replicates, but it is not a memory improvement.<n>Training on this task is a long chance plateau followed by a sharp escape.<n>Much of the published gain needs no architecture at all.
  arXiv  Detail & Related papers  (2026-07-02T18:54:23Z)
• Robust Dreamer: Deviation-Aware Latent Gaussian Memory for Action-Controlled AR Video Generation [89.70897512515477]
  Action-controlled image-to-video generation is a promising paradigm for interactive world simulation, where each control signal should elicit an immediate visual response.<n>Maintaining visual fidelity and 3D consistency over long autoregressive rollouts remains challenging.<n>Existing 3D-aware methods often suffer from catastrophic drift due to two impediments: information loss from textitLatent--RGB Cycling, and the training--inference gap induced by the textiterror-free hypothesis.<n>We present textbfRobust Dreamer, a memory-augmented framework built around how to
  arXiv  Detail & Related papers  (2026-05-29T05:21:33Z)
• Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning [19.83087496179494]
  Unlearning in large language models (LLMs) aims to remove harmful training data while preserving overall utility.<n>Existing methods often hallucinate, generate abnormal token sequences, or behave inconsistently, raising safety and trust concerns.<n>We propose a formal definition of unlearning honesty, which includes preserving both utility and honesty on retained knowledge.
  arXiv  Detail & Related papers  (2026-05-09T07:50:27Z)
• SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion [39.17638540496959]
  SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion) is a retain-set-free unlearning method built on a key insight.<n>High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence.<n> SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.
  arXiv  Detail & Related papers  (2026-05-08T09:25:18Z)
• Not Every Subject Should Stay: Machine Unlearning for Noisy Engagement Recognition [53.005382593686356]
  Engagement recognition datasets are typically subject-indexed and often contain noisy, subjective supervision.<n>We study this setting through subject-level machine unlearning as a post-hoc sanitization mechanism for engagement recognition.
  arXiv  Detail & Related papers  (2026-05-06T10:03:06Z)
• A Rubric-Supervised Critic from Sparse Real-World Outcomes [87.11204512676193]
  Real-world coding agents operate with humans in the loop, where success signals are typically noisy, delayed, and sparse.<n>We propose a process to learn a "critic" model from sparse and noisy interaction data, which can then be used both as a reward model for either RL-based training or inference-time scaling.
  arXiv  Detail & Related papers  (2026-03-04T07:23:54Z)
• Learning under noisy supervision is governed by a feedback-truth gap [0.0]
  We show that when feedback is absorbed faster than task structure can be evaluated, the learner will favor feedback over truth.<n>We test this prediction across neural networks trained with noisy labels (30 datasets, 2,700 runs), human probabilistic reversal learning (N = 292), and human reward/punishment learning with concurrent EEG (N = 25)<n>The gap is a fundamental constraint on learning under noisy supervision; its consequences depend on the regulation each system employs.
  arXiv  Detail & Related papers  (2026-02-18T19:50:56Z)
• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations [103.16279860448874]
  We propose an online reinforcement learning method using a novel binary retrieval-augmented reward (RAR)<n>For open-ended generation, binary RAR achieves a 39.3% reduction in hallucination rates.<n>In short-form question answering, the model learns abstention, strategically outputting "I don't know" when faced with insufficient parametric knowledge.
  arXiv  Detail & Related papers  (2025-10-20T16:45:43Z)
• Warp Consistency for Unsupervised Learning of Dense Correspondences [116.56251250853488]
  Key challenge in learning dense correspondences is lack of ground-truth matches for real image pairs. We propose Warp Consistency, an unsupervised learning objective for dense correspondence regression. Our approach sets a new state-of-the-art on several challenging benchmarks, including MegaDepth, RobotCar and TSS.
  arXiv  Detail & Related papers  (2021-04-07T17:58:22Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.