Investigating Compounding Prediction Errors in Learned Dynamics Models

• URL: http://arxiv.org/abs/2203.09637v1
• Date: Thu, 17 Mar 2022 22:24:38 GMT
• Title: Investigating Compounding Prediction Errors in Learned Dynamics Models
• Authors: Nathan Lambert, Kristofer Pister, Roberto Calandra
• Abstract summary: Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction.
• Score: 7.237751303770201
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurately predicting the consequences of agents' actions is a key prerequisite for planning in robotic control. Model-based reinforcement learning (MBRL) is one paradigm which relies on the iterative learning and prediction of state-action transitions to solve a task. Deep MBRL has become a popular candidate, using a neural network to learn a dynamics model that predicts with each pass from high-dimensional states to actions. These "one-step" predictions are known to become inaccurate over longer horizons of composed prediction - called the compounding error problem. Given the prevalence of the compounding error problem in MBRL and related fields of data-driven control, we set out to understand the properties of and conditions causing these long-horizon errors. In this paper, we explore the effects of subcomponents of a control problem on long term prediction error: including choosing a system, collecting data, and training a model. These detailed quantitative studies on simulated and real-world data show that the underlying dynamics of a system are the strongest factor determining the shape and magnitude of prediction error. Given a clearer understanding of compounding prediction error, researchers can implement new types of models beyond "one-step" that are more useful for control.

Related papers

• Mechanism learning: Reverse causal inference in the presence of multiple unknown confounding through front-door causal bootstrapping [0.8901073744693314]
  A major limitation of machine learning (ML) prediction models is that they recover associational, rather than causal, predictive relationships between variables. This paper proposes mechanism learning, a simple method which uses front-door causal bootstrapping to deconfound observational data. We test our method on fully synthetic, semi-synthetic and real-world datasets, demonstrating that it can discover reliable, unbiased, causal ML predictors.
  arXiv  Detail & Related papers  (2024-10-26T03:34:55Z)
• EANet: Expert Attention Network for Online Trajectory Prediction [5.600280639034753]
  Expert Attention Network is a complete online learning framework for trajectory prediction. We introduce expert attention, which adjusts the weights of different depths of network layers, avoiding the model updated slowly due to gradient problem. Furthermore, we propose a short-term motion trend kernel function which is sensitive to scenario change, allowing the model to respond quickly.
  arXiv  Detail & Related papers  (2023-09-11T07:09:40Z)
• Learning Residual Model of Model Predictive Control via Random Forests for Autonomous Driving [13.865293598486492]
  One major issue in predictive control (MPC) for autonomous driving is the contradiction between the system model's prediction and computation. This paper reformulates the MPC tracking accuracy as a program (QP) problem optimization as a program (QP) can effectively solve it.
  arXiv  Detail & Related papers  (2023-04-10T03:32:09Z)
• Predictable MDP Abstraction for Unsupervised Model-Based RL [93.91375268580806]
  We propose predictable MDP abstraction (PMA) Instead of training a predictive model on the original MDP, we train a model on a transformed MDP with a learned action space. We theoretically analyze PMA and empirically demonstrate that PMA leads to significant improvements over prior unsupervised model-based RL approaches.
  arXiv  Detail & Related papers  (2023-02-08T07:37:51Z)
• Real-to-Sim: Predicting Residual Errors of Robotic Systems with Sparse Data using a Learning-based Unscented Kalman Filter [65.93205328894608]
  We learn the residual errors between a dynamic and/or simulator model and the real robot. We show that with the learned residual errors, we can further close the reality gap between dynamic models, simulations, and actual hardware.
  arXiv  Detail & Related papers  (2022-09-07T15:15:12Z)
• On the Generalization and Adaption Performance of Causal Models [99.64022680811281]
  Differentiable causal discovery has proposed to factorize the data generating process into a set of modules. We study the generalization and adaption performance of such modular neural causal models. Our analysis shows that the modular neural causal models outperform other models on both zero and few-shot adaptation in low data regimes.
  arXiv  Detail & Related papers  (2022-06-09T17:12:32Z)
• Using Data Assimilation to Train a Hybrid Forecast System that Combines Machine-Learning and Knowledge-Based Components [52.77024349608834]
  We consider the problem of data-assisted forecasting of chaotic dynamical systems when the available data is noisy partial measurements. We show that by using partial measurements of the state of the dynamical system, we can train a machine learning model to improve predictions made by an imperfect knowledge-based model.
  arXiv  Detail & Related papers  (2021-02-15T19:56:48Z)
• Sufficiently Accurate Model Learning for Planning [119.80502738709937]
  This paper introduces the constrained Sufficiently Accurate model learning approach. It provides examples of such problems, and presents a theorem on how close some approximate solutions can be. The approximate solution quality will depend on the function parameterization, loss and constraint function smoothness, and the number of samples in model learning.
  arXiv  Detail & Related papers  (2021-02-11T16:27:31Z)
• Learning Accurate Long-term Dynamics for Model-based Reinforcement Learning [7.194382512848327]
  We propose a new parametrization to supervised learning on state-action data to stably predict at longer horizons. Our results in simulated and experimental robotic tasks show that our trajectory-based models yield significantly more accurate long term predictions.
  arXiv  Detail & Related papers  (2020-12-16T18:47:37Z)
• Bootstrapped model learning and error correction for planning with uncertainty in model-based RL [1.370633147306388]
  A natural aim is to learn a model that reflects accurately the dynamics of the environment. This paper explores the problem of model misspecification through uncertainty-aware reinforcement learning agents. We propose a bootstrapped multi-headed neural network that learns the distribution of future states and rewards.
  arXiv  Detail & Related papers  (2020-04-15T15:41:21Z)
• Dropout: Explicit Forms and Capacity Control [57.36692251815882]
  We investigate capacity control provided by dropout in various machine learning problems. In deep learning, we show that the data-dependent regularizer due to dropout directly controls the Rademacher complexity of the underlying class of deep neural networks. We evaluate our theoretical findings on real-world datasets, including MovieLens, MNIST, and Fashion-MNIST.
  arXiv  Detail & Related papers  (2020-03-06T19:10:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.