Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search

• URL: http://arxiv.org/abs/2411.09722v1
• Date: Thu, 14 Nov 2024 11:10:36 GMT
• Title: Iterative Batch Reinforcement Learning via Safe Diversified Model-based Policy Search
• Authors: Amna Najib, Stefan Depeweg, Phillip Swazinna,
• Abstract summary: Batch reinforcement learning enables policy learning without direct interaction with the environment during training. This approach is well-suited for high-risk and cost-intensive applications, such as industrial control. We present an algorithmic methodology for iterative batch reinforcement learning based on ensemble-based model-based policy search.
• Score: 2.0072624123275533
• License:
• Abstract: Batch reinforcement learning enables policy learning without direct interaction with the environment during training, relying exclusively on previously collected sets of interactions. This approach is, therefore, well-suited for high-risk and cost-intensive applications, such as industrial control. Learned policies are commonly restricted to act in a similar fashion as observed in the batch. In a real-world scenario, learned policies are deployed in the industrial system, inevitably leading to the collection of new data that can subsequently be added to the existing recording. The process of learning and deployment can thus take place multiple times throughout the lifespan of a system. In this work, we propose to exploit this iterative nature of applying offline reinforcement learning to guide learned policies towards efficient and informative data collection during deployment, leading to continuous improvement of learned policies while remaining within the support of collected data. We present an algorithmic methodology for iterative batch reinforcement learning based on ensemble-based model-based policy search, augmented with safety and, importantly, a diversity criterion.

Related papers

• Language-Conditioned Semantic Search-Based Policy for Robotic Manipulation Tasks [2.1332830068386217]
  We propose a language-conditioned semantic search-based method to produce an online search-based policy. Our approach surpasses the performance of the baselines on the CALVIN benchmark and exhibits strong zero-shot adaptation capabilities.
  arXiv  Detail & Related papers  (2023-12-10T16:17:00Z)
• Statistically Efficient Variance Reduction with Double Policy Estimation for Off-Policy Evaluation in Sequence-Modeled Reinforcement Learning [53.97273491846883]
  We propose DPE: an RL algorithm that blends offline sequence modeling and offline reinforcement learning with Double Policy Estimation. We validate our method in multiple tasks of OpenAI Gym with D4RL benchmarks.
  arXiv  Detail & Related papers  (2023-08-28T20:46:07Z)
• A Regularized Implicit Policy for Offline Reinforcement Learning [54.7427227775581]
  offline reinforcement learning enables learning from a fixed dataset, without further interactions with the environment. We propose a framework that supports learning a flexible yet well-regularized fully-implicit policy. Experiments and ablation study on the D4RL dataset validate our framework and the effectiveness of our algorithmic designs.
  arXiv  Detail & Related papers  (2022-02-19T20:22:04Z)
• Verified Probabilistic Policies for Deep Reinforcement Learning [6.85316573653194]
  We tackle the problem of verifying probabilistic policies for deep reinforcement learning. We propose an abstraction approach, based on interval Markov decision processes, that yields guarantees on a policy's execution. We present techniques to build and solve these models using abstract interpretation, mixed-integer linear programming, entropy-based refinement and probabilistic model checking.
  arXiv  Detail & Related papers  (2022-01-10T23:55:04Z)
• Constructing a Good Behavior Basis for Transfer using Generalized Policy Updates [63.58053355357644]
  We study the problem of learning a good set of policies, so that when combined together, they can solve a wide variety of unseen reinforcement learning tasks. We show theoretically that having access to a specific set of diverse policies, which we call a set of independent policies, can allow for instantaneously achieving high-level performance.
  arXiv  Detail & Related papers  (2021-12-30T12:20:46Z)
• Behavior Constraining in Weight Space for Offline Reinforcement Learning [2.7184068098378855]
  In offline reinforcement learning, a policy needs to be learned from a single dataset. We propose a new algorithm, which constrains the policy directly in its weight space instead, and demonstrate its effectiveness in experiments.
  arXiv  Detail & Related papers  (2021-07-12T14:50:50Z)
• Goal-Conditioned Reinforcement Learning with Imagined Subgoals [89.67840168694259]
  We propose to incorporate imagined subgoals into policy learning to facilitate learning of complex tasks. Imagined subgoals are predicted by a separate high-level policy, which is trained simultaneously with the policy and its critic. We evaluate our approach on complex robotic navigation and manipulation tasks and show that it outperforms existing methods by a large margin.
  arXiv  Detail & Related papers  (2021-07-01T15:30:59Z)
• Contextual Policy Transfer in Reinforcement Learning Domains via Deep Mixtures-of-Experts [24.489002406693128]
  We introduce a novel mixture-of-experts formulation for learning state-dependent beliefs over source task dynamics. We show how this model can be incorporated into standard policy reuse frameworks.
  arXiv  Detail & Related papers  (2020-02-29T07:58:36Z)
• Efficient Deep Reinforcement Learning via Adaptive Policy Transfer [50.51637231309424]
  Policy Transfer Framework (PTF) is proposed to accelerate Reinforcement Learning (RL) Our framework learns when and which source policy is the best to reuse for the target policy and when to terminate it. Experimental results show it significantly accelerates the learning process and surpasses state-of-the-art policy transfer methods.
  arXiv  Detail & Related papers  (2020-02-19T07:30:57Z)
• Reward-Conditioned Policies [100.64167842905069]
  imitation learning requires near-optimal expert data. Can we learn effective policies via supervised learning without demonstrations? We show how such an approach can be derived as a principled method for policy search.
  arXiv  Detail & Related papers  (2019-12-31T18:07:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.