World Models for Anomaly Detection during Model-Based Reinforcement Learning Inference

• URL: http://arxiv.org/abs/2503.02552v1
• Date: Tue, 04 Mar 2025 12:25:01 GMT
• Title: World Models for Anomaly Detection during Model-Based Reinforcement Learning Inference
• Authors: Fabian Domberg, Georg Schildbach,
• Abstract summary: Learning-based controllers are often purposefully kept out of real-world applications due to concerns about their safety and reliability.<n>We explore how state-of-the-art world models in Model-Based Reinforcement Learning can be utilized beyond the training phase to ensure a deployed policy only operates within regions of the state-space it is sufficiently familiar with.
• Score: 3.591122855617648
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Learning-based controllers are often purposefully kept out of real-world applications due to concerns about their safety and reliability. We explore how state-of-the-art world models in Model-Based Reinforcement Learning can be utilized beyond the training phase to ensure a deployed policy only operates within regions of the state-space it is sufficiently familiar with. This is achieved by continuously monitoring discrepancies between a world model's predictions and observed system behavior during inference. It allows for triggering appropriate measures, such as an emergency stop, once an error threshold is surpassed. This does not require any task-specific knowledge and is thus universally applicable. Simulated experiments on established robot control tasks show the effectiveness of this method, recognizing changes in local robot geometry and global gravitational magnitude. Real-world experiments using an agile quadcopter further demonstrate the benefits of this approach by detecting unexpected forces acting on the vehicle. These results indicate how even in new and adverse conditions, safe and reliable operation of otherwise unpredictable learning-based controllers can be achieved.

Related papers

• Robotic World Model: A Neural Network Simulator for Robust Policy Optimization in Robotics [50.191655141020505]
  We introduce a novel framework for learning world models.<n>By providing a scalable and robust framework, we pave the way for adaptive and efficient robotic systems in real-world applications.
  arXiv  Detail & Related papers  (2025-01-17T10:39:09Z)
• Mitigating Covariate Shift in Imitation Learning for Autonomous Vehicles Using Latent Space Generative World Models [60.87795376541144]
  A world model is a neural network capable of predicting an agent's next state given past states and actions. During end-to-end training, our policy learns how to recover from errors by aligning with states observed in human demonstrations. We present qualitative and quantitative results, demonstrating significant improvements upon prior state of the art in closed-loop testing.
  arXiv  Detail & Related papers  (2024-09-25T06:48:25Z)
• Behavioral Cloning Models Reality Check for Autonomous Driving [5.021331908103135]
  This paper presents the real-world validation of state-of-the-art perception systems that utilize Behavior Cloning (BC) for lateral control. The dataset was collected using a scaled research vehicle and tested on various track setups. Experimental results demonstrate that these methods predict steering angles with low error margins in real-time.
  arXiv  Detail & Related papers  (2024-09-11T12:19:38Z)
• Zero-shot Safety Prediction for Autonomous Robots with Foundation World Models [0.12499537119440243]
  A world model creates a surrogate world to train a controller and predict safety violations by learning the internal dynamic model of systems. We propose foundation world models that embed observations into meaningful and causally latent representations. This enables the surrogate dynamics to directly predict causal future states by leveraging a training-free large language model.
  arXiv  Detail & Related papers  (2024-03-30T20:03:49Z)
• Towards a Framework for Deep Learning Certification in Safety-Critical Applications Using Inherently Safe Design and Run-Time Error Detection [0.0]
  We consider real-world problems arising in aviation and other safety-critical areas, and investigate their requirements for a certified model. We establish a new framework towards deep learning certification based on (i) inherently safe design, and (ii) run-time error detection.
  arXiv  Detail & Related papers  (2024-03-12T11:38:45Z)
• Self-consistent Validation for Machine Learning Electronic Structure [81.54661501506185]
  Method integrates machine learning with self-consistent field methods to achieve both low validation cost and interpret-ability. This, in turn, enables exploration of the model's ability with active learning and instills confidence in its integration into real-world studies.
  arXiv  Detail & Related papers  (2024-02-15T18:41:35Z)
• Model-Based Runtime Monitoring with Interactive Imitation Learning [30.70994322652745]
  This work aims to endow a robot with the ability to monitor and detect errors during task execution. We introduce a model-based runtime monitoring algorithm that learns from deployment data to detect system anomalies and anticipate failures. Our method outperforms the baselines across system-level and unit-test metrics, with 23% and 40% higher success rates in simulation and on physical hardware.
  arXiv  Detail & Related papers  (2023-10-26T16:45:44Z)
• Model-Based Reinforcement Learning with Isolated Imaginations [61.67183143982074]
  We propose Iso-Dream++, a model-based reinforcement learning approach. We perform policy optimization based on the decoupled latent imaginations. This enables long-horizon visuomotor control tasks to benefit from isolating mixed dynamics sources in the wild.
  arXiv  Detail & Related papers  (2023-03-27T02:55:56Z)
• Guided Uncertainty-Aware Policy Optimization: Combining Learning and Model-Based Strategies for Sample-Efficient Policy Learning [75.56839075060819]
  Traditional robotic approaches rely on an accurate model of the environment, a detailed description of how to perform the task, and a robust perception system to keep track of the current state. reinforcement learning approaches can operate directly from raw sensory inputs with only a reward signal to describe the task, but are extremely sample-inefficient and brittle. In this work, we combine the strengths of model-based methods with the flexibility of learning-based methods to obtain a general method that is able to overcome inaccuracies in the robotics perception/actuation pipeline.
  arXiv  Detail & Related papers  (2020-05-21T19:47:05Z)
• From Simulation to Real World Maneuver Execution using Deep Reinforcement Learning [69.23334811890919]
  Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios. This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets. We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
  arXiv  Detail & Related papers  (2020-05-13T14:22:20Z)
• Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learning [109.77163932886413]
  We show how to adapt vision-based robotic manipulation policies to new variations by fine-tuning via off-policy reinforcement learning. This adaptation uses less than 0.2% of the data necessary to learn the task from scratch. We find that our approach of adapting pre-trained policies leads to substantial performance gains over the course of fine-tuning.
  arXiv  Detail & Related papers  (2020-04-21T17:57:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.