Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

• URL: http://arxiv.org/abs/2011.01734v1
• Date: Tue, 3 Nov 2020 14:37:53 GMT
• Title: Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning
• Authors: Michael Lutter, Johannes Silberbauer, Joe Watson, Jan Peters
• Abstract summary: A limitation of model-based reinforcement learning is the exploitation of errors in the learned models. We show that physics-based models can be beneficial compared to high-capacity function approximators if the mechanical structure is known.
• Score: 34.558299591341
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A limitation of model-based reinforcement learning (MBRL) is the exploitation of errors in the learned models. Black-box models can fit complex dynamics with high fidelity, but their behavior is undefined outside of the data distribution.Physics-based models are better at extrapolating, due to the general validity of their informed structure, but underfit in the real world due to the presence of unmodeled phenomena. In this work, we demonstrate experimentally that for the offline model-based reinforcement learning setting, physics-based models can be beneficial compared to high-capacity function approximators if the mechanical structure is known. Physics-based models can learn to perform the ball in a cup (BiC) task on a physical manipulator using only 4 minutes of sampled data using offline MBRL. We find that black-box models consistently produce unviable policies for BiC as all predicted trajectories diverge to physically impossible state, despite having access to more data than the physics-based model. In addition, we generalize the approach of physics parameter identification from modeling holonomic multi-body systems to systems with nonholonomic dynamics using end-to-end automatic differentiation. Videos: https://sites.google.com/view/ball-in-a-cup-in-4-minutes/

Related papers

• Learning to Walk from Three Minutes of Real-World Data with Semi-structured Dynamics Models [9.318262213262866]
  We introduce a novel framework for learning semi-structured dynamics models for contact-rich systems. We make accurate long-horizon predictions with substantially less data than prior methods. We validate our approach on a real-world Unitree Go1 quadruped robot.
  arXiv  Detail & Related papers  (2024-10-11T18:11:21Z)
• Exploring Model Transferability through the Lens of Potential Energy [78.60851825944212]
  Transfer learning has become crucial in computer vision tasks due to the vast availability of pre-trained deep learning models. Existing methods for measuring the transferability of pre-trained models rely on statistical correlations between encoded static features and task labels. We present an insightful physics-inspired approach named PED to address these challenges.
  arXiv  Detail & Related papers  (2023-08-29T07:15:57Z)
• Differentiable physics-enabled closure modeling for Burgers' turbulence [0.0]
  We discuss an approach using the differentiable physics paradigm that combines known physics with machine learning to develop closure models for turbulence problems. We train a series of models that incorporate varying degrees of physical assumptions on an a posteriori loss function to test the efficacy of models. We find that constraining models with inductive biases in the form of partial differential equations that contain known physics or existing closure approaches produces highly data-efficient, accurate, and generalizable models.
  arXiv  Detail & Related papers  (2022-09-23T14:38:01Z)
• Human Trajectory Prediction via Neural Social Physics [63.62824628085961]
  Trajectory prediction has been widely pursued in many fields, and many model-based and model-free methods have been explored. We propose a new method combining both methodologies based on a new Neural Differential Equation model. Our new model (Neural Social Physics or NSP) is a deep neural network within which we use an explicit physics model with learnable parameters.
  arXiv  Detail & Related papers  (2022-07-21T12:11:18Z)
• Gradient-Based Trajectory Optimization With Learned Dynamics [80.41791191022139]
  We use machine learning techniques to learn a differentiable dynamics model of the system from data. We show that a neural network can model highly nonlinear behaviors accurately for large time horizons. In our hardware experiments, we demonstrate that our learned model can represent complex dynamics for both the Spot and Radio-controlled (RC) car.
  arXiv  Detail & Related papers  (2022-04-09T22:07:34Z)
• Automated Dissipation Control for Turbulence Simulation with Shell Models [1.675857332621569]
  The application of machine learning (ML) techniques, especially neural networks, has seen tremendous success at processing images and language. In this work we construct a strongly simplified representation of turbulence by using the Gledzer-Ohkitani-Yamada shell model. We propose an approach that aims to reconstruct statistical properties of turbulence such as the self-similar inertial-range scaling.
  arXiv  Detail & Related papers  (2022-01-07T15:03:52Z)
• Which priors matter? Benchmarking models for learning latent dynamics [70.88999063639146]
  Several methods have proposed to integrate priors from classical mechanics into machine learning models. We take a sober look at the current capabilities of these models. We find that the use of continuous and time-reversible dynamics benefits models of all classes.
  arXiv  Detail & Related papers  (2021-11-09T23:48:21Z)
• Physics-Integrated Variational Autoencoders for Robust and Interpretable Generative Modeling [86.9726984929758]
  We focus on the integration of incomplete physics models into deep generative models. We propose a VAE architecture in which a part of the latent space is grounded by physics. We demonstrate generative performance improvements over a set of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-25T20:28:52Z)
• Data-Efficient Learning for Complex and Real-Time Physical Problem Solving using Augmented Simulation [49.631034790080406]
  We present a task for navigating a marble to the center of a circular maze. We present a model that learns to move a marble in the complex environment within minutes of interacting with the real system.
  arXiv  Detail & Related papers  (2020-11-14T02:03:08Z)
• Modeling System Dynamics with Physics-Informed Neural Networks Based on Lagrangian Mechanics [3.214927790437842]
  Two main modeling approaches often fail to meet requirements: first principles methods suffer from high bias, whereas data-driven modeling tends to have high variance. We present physics-informed neural ordinary differential equations (PINODE), a hybrid model that combines the two modeling techniques to overcome the aforementioned problems. Our findings are of interest for model-based control and system identification of mechanical systems.
  arXiv  Detail & Related papers  (2020-05-29T15:10:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.