Model-Based Reinforcement Learning with Isolated Imaginations
- URL: http://arxiv.org/abs/2303.14889v2
- Date: Fri, 17 Nov 2023 11:18:51 GMT
- Title: Model-Based Reinforcement Learning with Isolated Imaginations
- Authors: Minting Pan and Xiangming Zhu and Yitao Zheng and Yunbo Wang and
Xiaokang Yang
- Abstract summary: We propose Iso-Dream++, a model-based reinforcement learning approach.
We perform policy optimization based on the decoupled latent imaginations.
This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild.
- Score: 61.67183143982074
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: World models learn the consequences of actions in vision-based interactive
systems. However, in practical scenarios like autonomous driving,
noncontrollable dynamics that are independent or sparsely dependent on action
signals often exist, making it challenging to learn effective world models. To
address this issue, we propose Iso-Dream++, a model-based reinforcement
learning approach that has two main contributions. First, we optimize the
inverse dynamics to encourage the world model to isolate controllable state
transitions from the mixed spatiotemporal variations of the environment.
Second, we perform policy optimization based on the decoupled latent
imaginations, where we roll out noncontrollable states into the future and
adaptively associate them with the current controllable state. This enables
long-horizon visuomotor control tasks to benefit from isolating mixed dynamics
sources in the wild, such as self-driving cars that can anticipate the movement
of other vehicles, thereby avoiding potential risks. On top of our previous
work, we further consider the sparse dependencies between controllable and
noncontrollable states, address the training collapse problem of state
decoupling, and validate our approach in transfer learning setups. Our
empirical study demonstrates that Iso-Dream++ outperforms existing
reinforcement learning models significantly on CARLA and DeepMind Control.
Related papers
- Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models [60.87795376541144]
A world model is a neural network capable of predicting an agent's next state given past states and actions.
During end-to-end training, our policy learns how to recover from errors by aligning with states observed in human demonstrations.
We present qualitative and quantitative results, demonstrating significant improvements upon prior state of the art in closed-loop testing.
arXiv Detail & Related papers (2024-09-25T06:48:25Z) - Towards Human-Like Driving: Active Inference in Autonomous Vehicle Control [0.5437298646956507]
This paper presents a novel approach to Autonomous Vehicle (AV) control through the application of active inference.
Active inference is a theory derived from neuroscience that conceptualizes the brain as a predictive machine.
Our method integrates active inference with deep learning to manage lateral control in AVs, enabling them to perform lane following maneuvers within a simulated urban environment.
arXiv Detail & Related papers (2024-07-10T14:08:27Z) - Exploring Model Transferability through the Lens of Potential Energy [78.60851825944212]
Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models.
Existing methods for measuring the transferability of pre-trained models rely on statistical correlations between encoded static features and task labels.
We present an insightful physics-inspired approach named PED to address these challenges.
arXiv Detail & Related papers (2023-08-29T07:15:57Z) - Predictive Experience Replay for Continual Visual Control and
Forecasting [62.06183102362871]
We present a new continual learning approach for visual dynamics modeling and explore its efficacy in visual control and forecasting.
We first propose the mixture world model that learns task-specific dynamics priors with a mixture of Gaussians, and then introduce a new training strategy to overcome catastrophic forgetting.
Our model remarkably outperforms the naive combinations of existing continual learning and visual RL algorithms on DeepMind Control and Meta-World benchmarks with continual visual control tasks.
arXiv Detail & Related papers (2023-03-12T05:08:03Z) - Isolating and Leveraging Controllable and Noncontrollable Visual
Dynamics in World Models [65.97707691164558]
We present Iso-Dream, which improves the Dream-to-Control framework in two aspects.
First, by optimizing inverse dynamics, we encourage world model to learn controllable and noncontrollable sources.
Second, we optimize the behavior of the agent on the decoupled latent imaginations of the world model.
arXiv Detail & Related papers (2022-05-27T08:07:39Z) - Dream to Explore: Adaptive Simulations for Autonomous Systems [3.0664963196464448]
We tackle the problem of learning to control dynamical systems by applying Bayesian nonparametric methods.
By employing Gaussian processes to discover latent world dynamics, we mitigate common data efficiency issues observed in reinforcement learning.
Our algorithm jointly learns a world model and policy by optimizing a variational lower bound of a log-likelihood.
arXiv Detail & Related papers (2021-10-27T04:27:28Z) - Cycle-Consistent World Models for Domain Independent Latent Imagination [0.0]
High costs and risks make it hard to train autonomous cars in the real world.
We propose a novel model-based reinforcement learning approach called Cycleconsistent World Models.
arXiv Detail & Related papers (2021-10-02T13:55:50Z) - GEM: Group Enhanced Model for Learning Dynamical Control Systems [78.56159072162103]
We build effective dynamical models that are amenable to sample-based learning.
We show that learning the dynamics on a Lie algebra vector space is more effective than learning a direct state transition model.
This work sheds light on a connection between learning of dynamics and Lie group properties, which opens doors for new research directions.
arXiv Detail & Related papers (2021-04-07T01:08:18Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.