Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewards

• URL: http://arxiv.org/abs/2403.19024v2
• Date: Wed, 8 May 2024 05:41:07 GMT
• Title: Exploiting Symmetry in Dynamics for Model-Based Reinforcement Learning with Asymmetric Rewards
• Authors: Yasin Sonmez, Neelay Junnarkar, Murat Arcak,
• Abstract summary: We introduce a technique for learning dynamics which, by construction, exhibit specified symmetries. We demonstrate through numerical experiments that the proposed method learns a more accurate dynamical model.
• Score: 0.6612847014373572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work in reinforcement learning has leveraged symmetries in the model to improve sample efficiency in training a policy. A commonly used simplifying assumption is that the dynamics and reward both exhibit the same symmetry. However, in many real-world environments, the dynamical model exhibits symmetry independent of the reward model: the reward may not satisfy the same symmetries as the dynamics. In this paper, we investigate scenarios where only the dynamics are assumed to exhibit symmetry, extending the scope of problems in reinforcement learning and learning in control theory where symmetry techniques can be applied. We use Cartan's moving frame method to introduce a technique for learning dynamics which, by construction, exhibit specified symmetries. We demonstrate through numerical experiments that the proposed method learns a more accurate dynamical model.

Related papers

• A Generative Model of Symmetry Transformations [44.87295754993983]
  We build a generative model that explicitly aims to capture the data's approximate symmetries. We empirically demonstrate its ability to capture symmetries under affine and color transformations.
  arXiv  Detail & Related papers  (2024-03-04T11:32:18Z)
• A Multi-Grained Symmetric Differential Equation Model for Learning Protein-Ligand Binding Dynamics [73.35846234413611]
  In drug discovery, molecular dynamics (MD) simulation provides a powerful tool for predicting binding affinities, estimating transport properties, and exploring pocket sites. We propose NeuralMD, the first machine learning (ML) surrogate that can facilitate numerical MD and provide accurate simulations in protein-ligand binding dynamics. We demonstrate the efficiency and effectiveness of NeuralMD, achieving over 1K$times$ speedup compared to standard numerical MD simulations.
  arXiv  Detail & Related papers  (2024-01-26T09:35:17Z)
• A Unified Framework to Enforce, Discover, and Promote Symmetry in Machine Learning [5.1105250336911405]
  We provide a unifying theoretical and methodological framework for incorporating symmetry into machine learning models. We show that enforcing and discovering symmetry are linear-algebraic tasks that are dual with respect to the bilinear structure of the Lie derivative. We propose a novel way to promote symmetry by introducing a class of convex regularization functions based on the Lie derivative and nuclear norm relaxation.
  arXiv  Detail & Related papers  (2023-11-01T01:19:54Z)
• ${\rm E}(3)$-Equivariant Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learning [7.712824077083934]
  We focus on exploiting Euclidean symmetries inherent in certain cooperative multi-agent reinforcement learning problems. We design neural network architectures with symmetric constraints embedded as an inductive bias for multi-agent actor-critic methods.
  arXiv  Detail & Related papers  (2023-08-23T00:18:17Z)
• Capturing dynamical correlations using implicit neural representations [85.66456606776552]
  We develop an artificial intelligence framework which combines a neural network trained to mimic simulated data from a model Hamiltonian with automatic differentiation to recover unknown parameters from experimental data. In doing so, we illustrate the ability to build and train a differentiable model only once, which then can be applied in real-time to multi-dimensional scattering data.
  arXiv  Detail & Related papers  (2023-04-08T07:55:36Z)
• On discrete symmetries of robotics systems: A group-theoretic and data-driven analysis [38.92081817503126]
  We study discrete morphological symmetries of dynamical systems. These symmetries arise from the presence of one or more planes/axis of symmetry in the system's morphology. We exploit these symmetries using data augmentation and $G$-equivariant neural networks.
  arXiv  Detail & Related papers  (2023-02-21T04:10:16Z)
• The Surprising Effectiveness of Equivariant Models in Domains with Latent Symmetry [6.716931832076628]
  We show that imposing symmetry constraints that do not exactly match the domain symmetry is very helpful in learning the true symmetry in the environment. We demonstrate that an equivariant model can significantly outperform non-equivariant methods on domains with latent symmetries both in supervised learning and in reinforcement learning for robotic manipulation and control problems.
  arXiv  Detail & Related papers  (2022-11-16T21:51:55Z)
• On the Importance of Asymmetry for Siamese Representation Learning [53.86929387179092]
  Siamese networks are conceptually symmetric with two parallel encoders. We study the importance of asymmetry by explicitly distinguishing the two encoders within the network. We find the improvements from asymmetric designs generalize well to longer training schedules, multiple other frameworks and newer backbones.
  arXiv  Detail & Related papers  (2022-04-01T17:57:24Z)
• Learning continuous models for continuous physics [94.42705784823997]
  We develop a test based on numerical analysis theory to validate machine learning models for science and engineering applications. Our results illustrate how principled numerical analysis methods can be coupled with existing ML training/testing methodologies to validate models for science and engineering applications.
  arXiv  Detail & Related papers  (2022-02-17T07:56:46Z)
• A simple equivariant machine learning method for dynamics based on scalars [7.224406794844708]
  We show that the Scalars method outperforms state-of-the-art approaches for learning the properties of physical systems with symmetries. Because the method incorporates the fundamental symmetries, we expect it to generalize to different settings, such as changes in the force laws in the system.
  arXiv  Detail & Related papers  (2021-10-07T19:36:09Z)
• GELATO: Geometrically Enriched Latent Model for Offline Reinforcement Learning [54.291331971813364]
  offline reinforcement learning approaches can be divided into proximal and uncertainty-aware methods. In this work, we demonstrate the benefit of combining the two in a latent variational model. Our proposed metrics measure both the quality of out of distribution samples as well as the discrepancy of examples in the data.
  arXiv  Detail & Related papers  (2021-02-22T19:42:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.