Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective

• URL: http://arxiv.org/abs/2502.11480v1
• Date: Mon, 17 Feb 2025 06:34:58 GMT
• Title: Enhancing Offline Model-Based RL via Active Model Selection: A Bayesian Optimization Perspective
• Authors: Yu-Wei Yang, Yun-Ming Chan, Wei Hung, Xi Liu, Ping-Chun Hsieh,
• Abstract summary: offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data. We propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget. We show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only $1%$-$2.5%$ of offline training data.
• Score: 11.20804263996665
• License:
• Abstract: Offline model-based reinforcement learning (MBRL) serves as a competitive framework that can learn well-performing policies solely from pre-collected data with the help of learned dynamics models. To fully unleash the power of offline MBRL, model selection plays a pivotal role in determining the dynamics model utilized for downstream policy learning. However, offline MBRL conventionally relies on validation or off-policy evaluation, which are rather inaccurate due to the inherent distribution shift in offline RL. To tackle this, we propose BOMS, an active model selection framework that enhances model selection in offline MBRL with only a small online interaction budget, through the lens of Bayesian optimization (BO). Specifically, we recast model selection as BO and enable probabilistic inference in BOMS by proposing a novel model-induced kernel, which is theoretically grounded and computationally efficient. Through extensive experiments, we show that BOMS improves over the baseline methods with a small amount of online interaction comparable to only $1\%$-$2.5\%$ of offline training data on various RL tasks.

Related papers

• M$^3$PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model [14.779390462893298]
  We propose using Model Predictive Control (MPC) at test time to leverage the model's own predictive capability to guide its action selection. MPC significantly improves the decision-making performance of a pretrained trajectory model without any additional parameter training. Our framework can be adapted to Offline to Online (O2O) RL and Goal Reaching RL.
  arXiv  Detail & Related papers  (2024-12-07T14:44:22Z)
• Bayes Adaptive Monte Carlo Tree Search for Offline Model-based Reinforcement Learning [5.663006149337036]
  offline model-based reinforcement learning (MBRL) is a powerful approach for data-driven decision-making and control. There could be various MDPs that behave identically on the offline dataset and so dealing with the uncertainty about the true MDP can be challenging. We introduce a novel Bayes Adaptive Monte-Carlo planning algorithm capable of solving BAMDPs in continuous state and action spaces.
  arXiv  Detail & Related papers  (2024-10-15T03:36:43Z)
• Offline Model-Based Reinforcement Learning with Anti-Exploration [0.0]
  We present Morse Model-based offline RL (MoMo), which extends the anti-exploration paradigm found in offline model-free RL. MoMo performs offline reinforcement learning using an anti-exploration bonus to counteract value overestimation. The latter outperforms prior model-based and model-free baselines on the majority of D4RL datasets tested.
  arXiv  Detail & Related papers  (2024-08-20T10:29:21Z)
• When to Update Your Model: Constrained Model-based Reinforcement Learning [50.74369835934703]
  We propose a novel and general theoretical scheme for a non-decreasing performance guarantee of model-based RL (MBRL) Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.
  arXiv  Detail & Related papers  (2022-10-15T17:57:43Z)
• A Unified Framework for Alternating Offline Model Training and Policy Learning [62.19209005400561]
  In offline model-based reinforcement learning, we learn a dynamic model from historically collected data, and utilize the learned model and fixed datasets for policy learning. We develop an iterative offline MBRL framework, where we maximize a lower bound of the true expected return. With the proposed unified model-policy learning framework, we achieve competitive performance on a wide range of continuous-control offline reinforcement learning datasets.
  arXiv  Detail & Related papers  (2022-10-12T04:58:51Z)
• Double Check Your State Before Trusting It: Confidence-Aware Bidirectional Offline Model-Based Imagination [31.805991958408438]
  We propose to augment the offline dataset by using trained bidirectional dynamics models and rollout policies with double check. Our method, confidence-aware bidirectional offline model-based imagination, generates reliable samples and can be combined with any model-free offline RL method.
  arXiv  Detail & Related papers  (2022-06-16T08:00:44Z)
• Pessimistic Model Selection for Offline Deep Reinforcement Learning [56.282483586473816]
  Deep Reinforcement Learning (DRL) has demonstrated great potentials in solving sequential decision making problems in many applications. One main barrier is the over-fitting issue that leads to poor generalizability of the policy learned by DRL. We propose a pessimistic model selection (PMS) approach for offline DRL with a theoretical guarantee.
  arXiv  Detail & Related papers  (2021-11-29T06:29:49Z)
• Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL [82.93243616342275]
  We introduce Offline Model-based RL with Adaptive Behavioral Priors (MABE) MABE is based on the finding that dynamics models, which support within-domain generalization, and behavioral priors, which support cross-domain generalization, are complementary. In experiments that require cross-domain generalization, we find that MABE outperforms prior methods.
  arXiv  Detail & Related papers  (2021-06-16T20:48:49Z)
• MOPO: Model-based Offline Policy Optimization [183.6449600580806]
  offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. We show that an existing model-based RL algorithm already produces significant gains in the offline setting. We propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics.
  arXiv  Detail & Related papers  (2020-05-27T08:46:41Z)
• MOReL : Model-Based Offline Reinforcement Learning [49.30091375141527]
  In offline reinforcement learning (RL), the goal is to learn a highly rewarding policy based solely on a dataset of historical interactions with the environment. We present MOReL, an algorithmic framework for model-based offline RL. We show that MOReL matches or exceeds state-of-the-art results in widely studied offline RL benchmarks.
  arXiv  Detail & Related papers  (2020-05-12T17:52:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.