Online Baum-Welch algorithm for Hierarchical Imitation Learning

• URL: http://arxiv.org/abs/2103.12197v1
• Date: Mon, 22 Mar 2021 22:03:25 GMT
• Title: Online Baum-Welch algorithm for Hierarchical Imitation Learning
• Authors: Vittorio Giammarino and Ioannis Ch. Paschalidis
• Abstract summary: We propose an online algorithm to perform hierarchical imitation learning in the options framework. We show that this approach works well in both discrete and continuous environments.
• Score: 7.271970309320002
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The options framework for hierarchical reinforcement learning has increased its popularity in recent years and has made improvements in tackling the scalability problem in reinforcement learning. Yet, most of these recent successes are linked with a proper options initialization or discovery. When an expert is available, the options discovery problem can be addressed by learning an options-type hierarchical policy directly from expert demonstrations. This problem is referred to as hierarchical imitation learning and can be handled as an inference problem in a Hidden Markov Model, which is done via an Expectation-Maximization type algorithm. In this work, we propose a novel online algorithm to perform hierarchical imitation learning in the options framework. Further, we discuss the benefits of such an algorithm and compare it with its batch version in classical reinforcement learning benchmarks. We show that this approach works well in both discrete and continuous environments and, under certain conditions, it outperforms the batch version.

Related papers

• A Provably Efficient Option-Based Algorithm for both High-Level and Low-Level Learning [54.20447310988282]
  We present a meta-algorithm alternating between regret minimization algorithms instanced at different (high and low) temporal abstractions. At the higher level, we treat the problem as a Semi-Markov Decision Process (SMDP), with fixed low-level policies, while at a lower level, inner option policies are learned with a fixed high-level policy.
  arXiv  Detail & Related papers  (2024-06-21T13:17:33Z)
• ReconBoost: Boosting Can Achieve Modality Reconcilement [89.4377895465204]
  We study the modality-alternating learning paradigm to achieve reconcilement. We propose a new method called ReconBoost to update a fixed modality each time. We show that the proposed method resembles Friedman's Gradient-Boosting (GB) algorithm, where the updated learner can correct errors made by others.
  arXiv  Detail & Related papers  (2024-05-15T13:22:39Z)
• Multi-Task Off-Policy Learning from Bandit Feedback [54.96011624223482]
  We propose a hierarchical off-policy optimization algorithm (HierOPO), which estimates the parameters of the hierarchical model and then acts pessimistically with respect to them. We prove per-task bounds on the suboptimality of the learned policies, which show a clear improvement over not using the hierarchical model. Our theoretical and empirical results show a clear advantage of using the hierarchy over solving each task independently.
  arXiv  Detail & Related papers  (2022-12-09T08:26:27Z)
• Option-Aware Adversarial Inverse Reinforcement Learning for Robotic Control [44.77500987121531]
  Hierarchical Imitation Learning (HIL) has been proposed to recover highly-complex behaviors in long-horizon tasks from expert demonstrations. We develop a novel HIL algorithm based on Adversarial Inverse Reinforcement Learning. We also propose a Variational Autoencoder framework for learning with our objectives in an end-to-end fashion.
  arXiv  Detail & Related papers  (2022-10-05T00:28:26Z)
• Tree-Based Adaptive Model Learning [62.997667081978825]
  We extend the Kearns-Vazirani learning algorithm to handle systems that change over time. We present a new learning algorithm that can reuse and update previously learned behavior, implement it in the LearnLib library, and evaluate it on large examples.
  arXiv  Detail & Related papers  (2022-08-31T21:24:22Z)
• Meta Navigator: Search for a Good Adaptation Policy for Few-shot Learning [113.05118113697111]
  Few-shot learning aims to adapt knowledge learned from previous tasks to novel tasks with only a limited amount of labeled data. Research literature on few-shot learning exhibits great diversity, while different algorithms often excel at different few-shot learning scenarios. We present Meta Navigator, a framework that attempts to solve the limitation in few-shot learning by seeking a higher-level strategy.
  arXiv  Detail & Related papers  (2021-09-13T07:20:01Z)
• Adversarial Option-Aware Hierarchical Imitation Learning [89.92994158193237]
  We propose Option-GAIL, a novel method to learn skills at long horizon. The key idea of Option-GAIL is modeling the task hierarchy by options and train the policy via generative adversarial optimization. Experiments show that Option-GAIL outperforms other counterparts consistently across a variety of tasks.
  arXiv  Detail & Related papers  (2021-06-10T06:42:05Z)
• Provable Hierarchical Imitation Learning via EM [2.864550757598007]
  We consider learning an options-type hierarchical policy from expert demonstrations. We characterize the EM approach proposed by Daniel et al. We prove that the proposed algorithm converges with high probability to a norm ball around the true parameter.
  arXiv  Detail & Related papers  (2020-10-07T03:21:57Z)
• Reinforcement Learning for Variable Selection in a Branch and Bound Algorithm [0.10499611180329801]
  We leverage patterns in real-world instances to learn from scratch a new branching strategy optimised for a given problem. We propose FMSTS, a novel Reinforcement Learning approach specifically designed for this task.
  arXiv  Detail & Related papers  (2020-05-20T13:15:48Z)
• Unbiased Deep Reinforcement Learning: A General Training Framework for Existing and Future Algorithms [3.7050607140679026]
  We propose a novel training framework that is conceptually comprehensible and potentially easy to be generalized to all feasible algorithms for reinforcement learning. We employ Monte-carlo sampling to achieve raw data inputs, and train them in batch to achieve Markov decision process sequences. We propose several algorithms embedded with our new framework to deal with typical discrete and continuous scenarios.
  arXiv  Detail & Related papers  (2020-05-12T01:51:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.