Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction

• URL: http://arxiv.org/abs/2407.13364v1
• Date: Thu, 18 Jul 2024 10:15:51 GMT
• Title: Geometric Active Exploration in Markov Decision Processes: the Benefit of Abstraction
• Authors: Riccardo De Santi, Federico Arangath Joseph, Noah Liniger, Mirco Mutti, Andreas Krause,
• Abstract summary: We show how to use MDP homomorphisms formalism to exploit known geometric structures via abstraction. We also present the first analysis that formally captures the benefit of abstraction via homomorphisms on sample efficiency. We propose the Geometric Active Exploration (GAE) algorithm, which we analyse theoretically and experimentally in environments motivated by problems in scientific discovery.
• Score: 41.22779249609767
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: How can a scientist use a Reinforcement Learning (RL) algorithm to design experiments over a dynamical system's state space? In the case of finite and Markovian systems, an area called Active Exploration (AE) relaxes the optimization problem of experiments design into Convex RL, a generalization of RL admitting a wider notion of reward. Unfortunately, this framework is currently not scalable and the potential of AE is hindered by the vastness of experiment spaces typical of scientific discovery applications. However, these spaces are often endowed with natural geometries, e.g., permutation invariance in molecular design, that an agent could leverage to improve the statistical and computational efficiency of AE. To achieve this, we bridge AE and MDP homomorphisms, which offer a way to exploit known geometric structures via abstraction. Towards this goal, we make two fundamental contributions: we extend MDP homomorphisms formalism to Convex RL, and we present, to the best of our knowledge, the first analysis that formally captures the benefit of abstraction via homomorphisms on sample efficiency. Ultimately, we propose the Geometric Active Exploration (GAE) algorithm, which we analyse theoretically and experimentally in environments motivated by problems in scientific discovery.

Related papers

• Spherinator and HiPSter: Representation Learning for Unbiased Knowledge Discovery from Simulations [0.0]
  We describe a new, unbiased, and machine learning based approach to obtain useful scientific insights from a broad range of simulations. Our concept is based on applying nonlinear dimensionality reduction to learn compact representations of the data in a low-dimensional space. We present a prototype using a rotational invariant hyperspherical variational convolutional autoencoder, utilizing a power distribution in the latent space, and trained on galaxies from IllustrisTNG simulation.
  arXiv  Detail & Related papers  (2024-06-06T07:34:58Z)
• LLM and Simulation as Bilevel Optimizers: A New Paradigm to Advance Physical Scientific Discovery [141.39722070734737]
  We propose to enhance the knowledge-driven, abstract reasoning abilities of Large Language Models with the computational strength of simulations. We introduce Scientific Generative Agent (SGA), a bilevel optimization framework. We conduct experiments to demonstrate our framework's efficacy in law discovery and molecular design.
  arXiv  Detail & Related papers  (2024-05-16T03:04:10Z)
• Constrained Exploration via Reflected Replica Exchange Stochastic Gradient Langevin Dynamics [10.290462113848054]
  ReSGLD is an effective tool for non-vinquadatic learning tasks in large-scale datasets. We explore the role of the simulation efficiency in constrained multi-modal distributions and image classification.
  arXiv  Detail & Related papers  (2024-05-13T15:25:03Z)
• Exploiting Multiple Abstractions in Episodic RL via Reward Shaping [23.61187560936501]
  We consider a linear hierarchy of abstraction layers of the Markov Decision Process (MDP) underlying the target domain. We propose a novel form of Reward Shaping where the solution obtained at the abstract level is used to offer rewards to the more concrete MDP.
  arXiv  Detail & Related papers  (2023-02-28T13:22:29Z)
• Reinforcement Learning in Factored Action Spaces using Tensor Decompositions [92.05556163518999]
  We propose a novel solution for Reinforcement Learning (RL) in large, factored action spaces using tensor decompositions. We use cooperative multi-agent reinforcement learning scenario as the exemplary setting.
  arXiv  Detail & Related papers  (2021-10-27T15:49:52Z)
• Provable RL with Exogenous Distractors via Multistep Inverse Dynamics [85.52408288789164]
  Real-world applications of reinforcement learning (RL) require the agent to deal with high-dimensional observations such as those generated from a megapixel camera. Prior work has addressed such problems with representation learning, through which the agent can provably extract endogenous, latent state information from raw observations. However, such approaches can fail in the presence of temporally correlated noise in the observations.
  arXiv  Detail & Related papers  (2021-10-17T15:21:27Z)
• Discovering Latent Causal Variables via Mechanism Sparsity: A New Principle for Nonlinear ICA [81.4991350761909]
  Independent component analysis (ICA) refers to an ensemble of methods which formalize this goal and provide estimation procedure for practical application. We show that the latent variables can be recovered up to a permutation if one regularizes the latent mechanisms to be sparse.
  arXiv  Detail & Related papers  (2021-07-21T14:22:14Z)
• Geometric Entropic Exploration [52.67987687712534]
  We introduce a new algorithm that maximises the geometry-aware Shannon entropy of state-visits in both discrete and continuous domains. Our key theoretical contribution is casting geometry-aware MSVE exploration as a tractable problem of optimising a simple and novel noise-contrastive objective function. In our experiments, we show the efficiency of GEM in solving several RL problems with sparse rewards, compared against other deep RL exploration approaches.
  arXiv  Detail & Related papers  (2021-01-06T14:15:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.