Reinforcement Learning with Options and State Representation

• URL: http://arxiv.org/abs/2403.10855v2
• Date: Mon, 25 Mar 2024 16:07:24 GMT
• Title: Reinforcement Learning with Options and State Representation
• Authors: Ayoub Ghriss, Masashi Sugiyama, Alessandro Lazaric,
• Abstract summary: This thesis aims to explore the reinforcement learning field and build on existing methods to produce improved ones. It addresses such goals by decomposing learning tasks in a hierarchical fashion known as Hierarchical Reinforcement Learning.
• Score: 105.82346211739433
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The current thesis aims to explore the reinforcement learning field and build on existing methods to produce improved ones to tackle the problem of learning in high-dimensional and complex environments. It addresses such goals by decomposing learning tasks in a hierarchical fashion known as Hierarchical Reinforcement Learning. We start in the first chapter by getting familiar with the Markov Decision Process framework and presenting some of its recent techniques that the following chapters use. We then proceed to build our Hierarchical Policy learning as an answer to the limitations of a single primitive policy. The hierarchy is composed of a manager agent at the top and employee agents at the lower level. In the last chapter, which is the core of this thesis, we attempt to learn lower-level elements of the hierarchy independently of the manager level in what is known as the "Eigenoption". Based on the graph structure of the environment, Eigenoptions allow us to build agents that are aware of the geometric and dynamic properties of the environment. Their decision-making has a special property: it is invariant to symmetric transformations of the environment, allowing as a consequence to greatly reduce the complexity of the learning task.

Related papers

• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation. Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy. Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• Temporal Abstraction in Reinforcement Learning with Offline Data [8.370420807869321]
  We propose a framework by which an online hierarchical reinforcement learning algorithm can be trained on an offline dataset of transitions collected by an unknown behavior policy. We validate our method on Gym MuJoCo environments and robotic gripper block-stacking tasks in the standard as well as transfer and goal-conditioned settings.
  arXiv  Detail & Related papers  (2024-07-21T18:10:31Z)
• A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning [54.20447310988282]
  We present a meta-algorithm alternating between regret minimization algorithms instanced at different (high and low) temporal abstractions. At the higher level, we treat the problem as a Semi-Markov Decision Process (SMDP), with fixed low-level policies, while at a lower level, inner option policies are learned with a fixed high-level policy.
  arXiv  Detail & Related papers  (2024-06-21T13:17:33Z)
• I Know How: Combining Prior Policies to Solve New Tasks [17.214443593424498]
  Multi-Task Reinforcement Learning aims at developing agents that are able to continually evolve and adapt to new scenarios. Learning from scratch for each new task is not a viable or sustainable option. We propose a new framework, I Know How, which provides a common formalization.
  arXiv  Detail & Related papers  (2024-06-14T08:44:51Z)
• Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control [44.77500987121531]
  Hierarchical Imitation Learning (HIL) has been proposed to recover highly-complex behaviors in long-horizon tasks from expert demonstrations. We develop a novel HIL algorithm based on Adversarial Inverse Reinforcement Learning. We also propose a Variational Autoencoder framework for learning with our objectives in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-10-05T00:28:26Z)
• Policy Architectures for Compositional Generalization in Control [71.61675703776628]
  We introduce a framework for modeling entity-based compositional structure in tasks. Our policies are flexible and can be trained end-to-end without requiring any action primitives.
  arXiv  Detail & Related papers  (2022-03-10T06:44:24Z)
• Provable Hierarchy-Based Meta-Reinforcement Learning [50.17896588738377]
  We analyze HRL in the meta-RL setting, where learner learns latent hierarchical structure during meta-training for use in a downstream task. We provide "diversity conditions" which, together with a tractable optimism-based algorithm, guarantee sample-efficient recovery of this natural hierarchy. Our bounds incorporate common notions in HRL literature such as temporal and state/action abstractions, suggesting that our setting and analysis capture important features of HRL in practice.
  arXiv  Detail & Related papers  (2021-10-18T17:56:02Z)
• Attaining Interpretability in Reinforcement Learning via Hierarchical Primitive Composition [3.1078562713129765]
  We propose a novel hierarchical reinforcement learning algorithm that mitigates the aforementioned issues by decomposing the original task in a hierarchy. We show how the proposed scheme can be employed in practice by solving a pick and place task with a 6 DoF manipulator.
  arXiv  Detail & Related papers  (2021-10-05T05:59:31Z)
• Hierarchically Decoupled Imitation for Morphological Transfer [95.19299356298876]
  We show that transferring learned information from a morphologically simpler agent can massively improve the sample efficiency of a more complex one. First, we show that incentivizing a complex agent's low-level to imitate a simpler agent's low-level significantly improves zero-shot high-level transfer. Second, we show that KL-regularized training of the high level stabilizes learning and prevents mode-collapse.
  arXiv  Detail & Related papers  (2020-03-03T18:56:49Z)
• Learning Functionally Decomposed Hierarchies for Continuous Control Tasks with Path Planning [36.050432925402845]
  We present HiDe, a novel hierarchical reinforcement learning architecture that successfully solves long horizon control tasks. We experimentally show that our method generalizes across unseen test environments and can scale to 3x horizon length compared to both learning and non-learning based methods.
  arXiv  Detail & Related papers  (2020-02-14T10:19:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.