Provable Zero-Shot Generalization in Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2503.07988v1
• Date: Tue, 11 Mar 2025 02:44:32 GMT
• Title: Provable Zero-Shot Generalization in Offline Reinforcement Learning
• Authors: Zhiyong Wang, Chen Yang, John C. S. Lui, Dongruo Zhou,
• Abstract summary: We study offline reinforcement learning with zero-shot generalization property (ZSG)<n>Existing work showed that classical offline RL fails to generalize to new, unseen environments.<n>We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG.
• Score: 55.169228792596805
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we study offline reinforcement learning (RL) with zero-shot generalization property (ZSG), where the agent has access to an offline dataset including experiences from different environments, and the goal of the agent is to train a policy over the training environments which performs well on test environments without further interaction. Existing work showed that classical offline RL fails to generalize to new, unseen environments. We propose pessimistic empirical risk minimization (PERM) and pessimistic proximal policy optimization (PPPO), which leverage pessimistic policy evaluation to guide policy learning and enhance generalization. We show that both PERM and PPPO are capable of finding a near-optimal policy with ZSG. Our result serves as a first step in understanding the foundation of the generalization phenomenon in offline reinforcement learning.

Related papers

• Large Language Model driven Policy Exploration for Recommender Systems [50.70228564385797]
  offline RL policies trained on static user data are vulnerable to distribution shift when deployed in dynamic online environments.<n>Online RL-based RS also face challenges in production deployment due to the risks of exposing users to untrained or unstable policies.<n>Large Language Models (LLMs) offer a promising solution to mimic user objectives and preferences for pre-training policies offline.<n>We propose an Interaction-Augmented Learned Policy (iALP) that utilizes user preferences distilled from an LLM.
  arXiv  Detail & Related papers  (2025-01-23T16:37:44Z)
• Online Reinforcement Learning in Non-Stationary Context-Driven Environments [12.954992692713898]
  We study online reinforcement learning (RL) in non-stationary environments.<n>Online RL is challenging in such environments due to "catastrophic forgetting" (CF)<n>We present Locally Constrained Policy Optimization (LCPO), an online RL approach that combats CF by anchoring policy outputs on old experiences.
  arXiv  Detail & Related papers  (2023-02-04T15:31:19Z)
• Offline Reinforcement Learning with Closed-Form Policy Improvement Operators [88.54210578912554]
  Behavior constrained policy optimization has been demonstrated to be a successful paradigm for tackling Offline Reinforcement Learning. In this paper, we propose our closed-form policy improvement operators. We empirically demonstrate their effectiveness over state-of-the-art algorithms on the standard D4RL benchmark.
  arXiv  Detail & Related papers  (2022-11-29T06:29:26Z)
• On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardness [47.09873295916592]
  Generalization in Reinforcement Learning (RL) aims to learn an agent during training that generalizes to the target environment. This paper studies RL generalization from a theoretical aspect: how much can we expect pre-training over training environments to be helpful? When the interaction with the target environment is not allowed, we certify that the best we can obtain is a near-optimal policy in an average sense, and we design an algorithm that achieves this goal.
  arXiv  Detail & Related papers  (2022-10-19T10:58:24Z)
• Offline Reinforcement Learning with Instrumental Variables in Confounded Markov Decision Processes [93.61202366677526]
  We study the offline reinforcement learning (RL) in the face of unmeasured confounders. We propose various policy learning methods with the finite-sample suboptimality guarantee of finding the optimal in-class policy.
  arXiv  Detail & Related papers  (2022-09-18T22:03:55Z)
• Model-Based Offline Meta-Reinforcement Learning with Regularization [63.35040401948943]
  offline Meta-RL is emerging as a promising approach to address these challenges. MerPO learns a meta-model for efficient task structure inference and an informative meta-policy. We show that MerPO offers guaranteed improvement over both the behavior policy and the meta-policy.
  arXiv  Detail & Related papers  (2022-02-07T04:15:20Z)
• Curriculum Offline Imitation Learning [72.1015201041391]
  offline reinforcement learning tasks require the agent to learn from a pre-collected dataset with no further interactions with the environment. We propose textitCurriculum Offline Learning (COIL), which utilizes an experience picking strategy for imitating from adaptive neighboring policies with a higher return. On continuous control benchmarks, we compare COIL against both imitation-based and RL-based methods, showing that it not only avoids just learning a mediocre behavior on mixed datasets but is also even competitive with state-of-the-art offline RL methods.
  arXiv  Detail & Related papers  (2021-11-03T08:02:48Z)
• Multi-Objective SPIBB: Seldonian Offline Policy Improvement with Safety Constraints in Finite MDPs [71.47895794305883]
  We study the problem of Safe Policy Improvement (SPI) under constraints in the offline Reinforcement Learning setting. We present an SPI for this RL setting that takes into account the preferences of the algorithm's user for handling the trade-offs for different reward signals.
  arXiv  Detail & Related papers  (2021-05-31T21:04:21Z)
• Instance based Generalization in Reinforcement Learning [24.485597364200824]
  We analyze policy learning in the context of Partially Observable Markov Decision Processes (POMDPs) We prove that, independently of the exploration strategy, reusing instances introduces significant changes on the effective Markov dynamics the agent observes during training. We propose training a shared belief representation over an ensemble of specialized policies, from which we compute a consensus policy that is used for data collection, disallowing instance specific exploitation.
  arXiv  Detail & Related papers  (2020-11-02T16:19:44Z)
• Expert-Supervised Reinforcement Learning for Offline Policy Learning and Evaluation [21.703965401500913]
  We propose an Expert-Supervised RL (ESRL) framework which uses uncertainty quantification for offline policy learning. In particular, we have three contributions: 1) the method can learn safe and optimal policies through hypothesis testing, 2) ESRL allows for different levels of risk averse implementations tailored to the application context, and 3) we propose a way to interpret ESRL's policy at every state through posterior distributions.
  arXiv  Detail & Related papers  (2020-06-23T17:43:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.