All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning

• URL: http://arxiv.org/abs/2606.01363v1
• Date: Sun, 31 May 2026 17:34:22 GMT
• Title: All Models are Wrong, Knowing Where is Useful: On Model Uncertainty in Reinforcement Learning
• Abstract summary: We present a framework for dealing with inaccuracies of probabilistic models through targeted handling of uncertainty.<n>We present recent successes in learning directly on hardware and safe exploration, and discuss future directions for uncertainty-aware model-based reinforcement learning.
• Score: 16.42180755359194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model-based reinforcement learning (MBRL) infers information about the environment from a learned dynamics model and bears the potential to address open problems such as data efficient and safe learning in robotics. However, inaccuracies of the learned dynamics model are typically exploited by the agent, substantially hampering the capabilities of MBRL methods. We present a framework for dealing with inaccuracies of probabilistic models through targeted handling of uncertainty that effectively mitigates model exploitation. We present recent successes in learning directly on hardware and safe exploration, and discuss future directions for uncertainty-aware MBRL.

Related papers

• Offline Robotic World Model: Learning Robotic Policies without a Physics Simulator [50.191655141020505]
  Reinforcement Learning (RL) has demonstrated impressive capabilities in robotic control but remains challenging due to high sample complexity, safety concerns, and the sim-to-real gap.<n>We introduce Offline Robotic World Model (RWM-O), a model-based approach that explicitly estimates uncertainty to improve policy learning without reliance on a physics simulator.
  arXiv  Detail & Related papers  (2025-04-23T12:58:15Z)
• Reinforcement Learning for Machine Learning Model Deployment: Evaluating Multi-Armed Bandits in ML Ops Environments [0.0]
  We investigate whether reinforcement learning (RL)-based model management can manage deployment decisions more effectively.<n>Our approach enables more adaptive production environments by continuously evaluating deployed models and rolling back underperforming ones in real-time.<n>Our findings suggest that RL-based model management can improve automation, reduce reliance on manual interventions, and mitigate risks associated with post-deployment model failures.
  arXiv  Detail & Related papers  (2025-03-28T16:42:21Z)
• Differentiable Information Enhanced Model-Based Reinforcement Learning [48.820039382764]
  Differentiable environments have heralded new possibilities for learning control policies by offering rich differentiable information.<n>Model-based reinforcement learning (MBRL) methods exhibit the potential to effectively harness the power of differentiable information for recovering the underlying physical dynamics.<n>However, this presents two primary challenges: effectively utilizing differentiable information to 1) construct models with more accurate dynamic prediction and 2) enhance the stability of policy training.
  arXiv  Detail & Related papers  (2025-03-03T04:51:40Z)
• Self-Destructing Models: Increasing the Costs of Harmful Dual Uses of Foundation Models [103.71308117592963]
  We present an algorithm for training self-destructing models leveraging techniques from meta-learning and adversarial learning. In a small-scale experiment, we show MLAC can largely prevent a BERT-style model from being re-purposed to perform gender identification.
  arXiv  Detail & Related papers  (2022-11-27T21:43:45Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)
• A Unified Framework for Alternating Offline Model Training and Policy Learning [62.19209005400561]
  In offline model-based reinforcement learning, we learn a dynamic model from historically collected data, and utilize the learned model and fixed datasets for policy learning. We develop an iterative offline MBRL framework, where we maximize a lower bound of the true expected return. With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets.
  arXiv  Detail & Related papers  (2022-10-12T04:58:51Z)
• Should Models Be Accurate? [14.044354912031864]
  We focus our investigations on Dyna-style planning in a prediction setting. We introduce a meta-learning algorithm for training models with a focus on their usefulness to the learner instead of their accuracy in modelling the environment. Our experiments show that our algorithm enables faster learning than even using an accurate model built with domain-specific knowledge of the non-stationarity.
  arXiv  Detail & Related papers  (2022-05-22T04:23:54Z)
• Quantifying Multimodality in World Models [5.593667856320704]
  We propose new metrics for the detection and quantification of multimodal uncertainty in RL based World Models. The correct modelling & detection of uncertain future states lays the foundation for handling critical situations in a safe way.
  arXiv  Detail & Related papers  (2021-12-14T09:52:18Z)
• Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control [1.8275108630751844]
  We propose to leverage the prior knowledge of underlying physics of the environment, where the governing laws are (partially) known. By incorporating the prior information of the environment, the quality of the learned model can be notably improved.
  arXiv  Detail & Related papers  (2021-07-31T02:19:36Z)
• Model Embedding Model-Based Reinforcement Learning [4.566180616886624]
  Model-based reinforcement learning (MBRL) has shown its advantages in sample-efficiency over model-free reinforcement learning (MFRL) Despite the impressive results it achieves, it still faces a trade-off between the ease of data generation and model bias. We propose a simple and elegant model-embedding model-based reinforcement learning (MEMB) algorithm in the framework of the probabilistic reinforcement learning.
  arXiv  Detail & Related papers  (2020-06-16T15:10:28Z)
• Efficient Model-Based Reinforcement Learning through Optimistic Policy Search and Planning [93.1435980666675]
  We show how optimistic exploration can be easily combined with state-of-the-art reinforcement learning algorithms. Our experiments demonstrate that optimistic exploration significantly speeds-up learning when there are penalties on actions.
  arXiv  Detail & Related papers  (2020-06-15T18:37:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.