Adapting World Models with Latent-State Dynamics Residuals

• URL: http://arxiv.org/abs/2504.02252v1
• Date: Thu, 03 Apr 2025 03:41:30 GMT
• Title: Adapting World Models with Latent-State Dynamics Residuals
• Abstract summary: ReDRAW is a latent-state autoregressive world model pretrained in simulation and calibrated to target environments.<n>It enables RL agents to be optimized with imagined rollouts under corrected dynamics and then deployed in the real world.
• Score: 10.892848566977369
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Simulation-to-reality reinforcement learning (RL) faces the critical challenge of reconciling discrepancies between simulated and real-world dynamics, which can severely degrade agent performance. A promising approach involves learning corrections to simulator forward dynamics represented as a residual error function, however this operation is impractical with high-dimensional states such as images. To overcome this, we propose ReDRAW, a latent-state autoregressive world model pretrained in simulation and calibrated to target environments through residual corrections of latent-state dynamics rather than of explicit observed states. Using this adapted world model, ReDRAW enables RL agents to be optimized with imagined rollouts under corrected dynamics and then deployed in the real world. In multiple vision-based MuJoCo domains and a physical robot visual lane-following task, ReDRAW effectively models changes to dynamics and avoids overfitting in low data regimes where traditional transfer methods fail.

Related papers

• Sword: Style-Robust World Models as Simulators via Dynamic Latent Bootstrapping for VLA Policy Post-Training [19.013714390803052]
  One approach treats learned World Models as generative simulators, enabling policy optimization entirely within "imagination"<n>However, when deployed as simulators for specific environments such as the LIBERO benchmark, existing World Models often suffer from poor generalization and long-horizon error accumulation.<n>We propose Sword, a robust World Model framework, to mitigate these problems.
  arXiv  Detail & Related papers  (2026-05-08T05:54:33Z)
• ResWM: Residual-Action World Model for Visual RL [0.06999740786886534]
  Learning predictive world models from raw visual observations is a central challenge in reinforcement learning (RL)<n>We introduce the Residual-Action World Model (ResWM), a new framework that reformulates the control variable from absolute actions to residual actions.<n>ResWM produces more stable and energy-efficient action trajectories, a property critical for robotic systems deployed in real-world environments.
  arXiv  Detail & Related papers  (2026-03-11T11:27:08Z)
• Aligning Agentic World Models via Knowledgeable Experience Learning [68.85843641222186]
  We introduce WorldMind, a framework that constructs a symbolic World Knowledge Repository by synthesizing environmental feedback.<n>WorldMind achieves superior performance compared to baselines with remarkable cross-model and cross-environment transferability.
  arXiv  Detail & Related papers  (2026-01-19T17:33:31Z)
• Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling [31.74241286023207]
  Reinforcement learning (RL) is playing an increasingly important role in fields such as robotic control and autonomous driving.<n>The gap between simulation and the real environment remains a major obstacle to the practical deployment of RL.<n>We propose a latent space based approach to analyze the impact of simulation on real-world policy improvement.
  arXiv  Detail & Related papers  (2025-06-15T06:02:42Z)
• RIFT: Group-Relative RL Fine-Tuning for Realistic and Controllable Traffic Simulation [13.319344167881383]
  We introduce a dual-stage AV-centric simulation framework that conducts imitation learning pre-training in a data-driven simulator.<n>We then learn fine-tuning in a physics-based simulator to enhance style-level controllability.<n>In the fine-tuning stage, we propose RIFT, a novel group-relative RL fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-05-06T09:12:37Z)
• Pre-Trained Video Generative Models as World Simulators [59.546627730477454]
  We propose Dynamic World Simulation (DWS) to transform pre-trained video generative models into controllable world simulators.<n>To achieve precise alignment between conditioned actions and generated visual changes, we introduce a lightweight, universal action-conditioned module.<n> Experiments demonstrate that DWS can be versatilely applied to both diffusion and autoregressive transformer models.
  arXiv  Detail & Related papers  (2025-02-10T14:49:09Z)
• Towards Unraveling and Improving Generalization in World Models [29.54936027897926]
  This work aims to obtain a deep understanding of robustness and generalization capabilities of world models. We characterize the impact of latent representation errors on robustness and generalization. We propose a Jacobian regularization scheme to mitigate the compounding error propagation effects of non-zero drift.
  arXiv  Detail & Related papers  (2024-12-31T00:15:43Z)
• Towards Realistic Data Generation for Real-World Super-Resolution [58.99206459754721]
  RealDGen is an unsupervised learning data generation framework designed for real-world super-resolution.<n>We develop content and degradation extraction strategies, which are integrated into a novel content-degradation decoupled diffusion model.<n>Experiments demonstrate that RealDGen excels in generating large-scale, high-quality paired data that mirrors real-world degradations.
  arXiv  Detail & Related papers  (2024-06-11T13:34:57Z)
• ReCoRe: Regularized Contrastive Representation Learning of World Model [21.29132219042405]
  We present a world model that learns invariant features using contrastive unsupervised learning and an intervention-invariant regularizer. Our method outperforms current state-of-the-art model-based and model-free RL methods and significantly improves on out-of-distribution point navigation tasks evaluated on the iGibson benchmark.
  arXiv  Detail & Related papers  (2023-12-14T15:53:07Z)
• Reinforcement Learning with Human Feedback for Realistic Traffic Simulation [53.85002640149283]
  Key element of effective simulation is the incorporation of realistic traffic models that align with human knowledge. This study identifies two main challenges: capturing the nuances of human preferences on realism and the unification of diverse traffic simulation models.
  arXiv  Detail & Related papers  (2023-09-01T19:29:53Z)
• Predictive Experience Replay for Continual Visual Control and Forecasting [62.06183102362871]
  We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting. We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting. Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:08:03Z)
• Enhancing Deep Traffic Forecasting Models with Dynamic Regression [15.31488551912888]
  This paper introduces a dynamic regression (DR) framework to enhance existing traffic forecasting models by structured learning for the residual process. We evaluate the effectiveness of the proposed framework on deep traffic forecasting models using both speed and flow datasets.
  arXiv  Detail & Related papers  (2023-01-17T01:12:44Z)
• Dream to Explore: Adaptive Simulations for Autonomous Systems [3.0664963196464448]
  We tackle the problem of learning to control dynamical systems by applying Bayesian nonparametric methods. By employing Gaussian processes to discover latent world dynamics, we mitigate common data efficiency issues observed in reinforcement learning. Our algorithm jointly learns a world model and policy by optimizing a variational lower bound of a log-likelihood.
  arXiv  Detail & Related papers  (2021-10-27T04:27:28Z)
• Autoregressive Dynamics Models for Offline Policy Evaluation and Optimization [60.73540999409032]
  We show that expressive autoregressive dynamics models generate different dimensions of the next state and reward sequentially conditioned on previous dimensions. We also show that autoregressive dynamics models are useful for offline policy optimization by serving as a way to enrich the replay buffer.
  arXiv  Detail & Related papers  (2021-04-28T16:48:44Z)
• Learning to Reweight Imaginary Transitions for Model-Based Reinforcement Learning [58.66067369294337]
  When the model is inaccurate or biased, imaginary trajectories may be deleterious for training the action-value and policy functions. We adaptively reweight the imaginary transitions, so as to reduce the negative effects of poorly generated trajectories. Our method outperforms state-of-the-art model-based and model-free RL algorithms on multiple tasks.
  arXiv  Detail & Related papers  (2021-04-09T03:13:35Z)
• Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning [124.9856253431878]
  We decompose the task of learning a global dynamics model into two stages: (a) learning a context latent vector that captures the local dynamics, then (b) predicting the next state conditioned on it. In order to encode dynamics-specific information into the context latent vector, we introduce a novel loss function that encourages the context latent vector to be useful for predicting both forward and backward dynamics. The proposed method achieves superior generalization ability across various simulated robotics and control tasks, compared to existing RL schemes.
  arXiv  Detail & Related papers  (2020-05-14T08:10:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.