Offline-Online Reinforcement Learning for Linear Mixture MDPs

• URL: http://arxiv.org/abs/2604.11994v1
• Date: Mon, 13 Apr 2026 19:34:30 GMT
• Title: Offline-Online Reinforcement Learning for Linear Mixture MDPs
• Abstract summary: We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift.<n>In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment.<n>We propose an algorithm that adaptively leverages offline data.
• Score: 0.5538445885586951
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study offline-online reinforcement learning in linear mixture Markov decision processes (MDPs) under environment shift. In the offline phase, data are collected by an unknown behavior policy and may come from a mismatched environment, while in the online phase the learner interacts with the target environment. We propose an algorithm that adaptively leverages offline data. When the offline data are informative, either due to sufficient coverage or small environment shift, the algorithm provably improves over purely online learning. When the offline data are uninformative, it safely ignores them and matches the online-only performance. We establish regret upper bounds that explicitly characterize when offline data are beneficial, together with nearly matching lower bounds. Numerical experiments further corroborate our theoretical findings.

Related papers

• Offline-to-Online Learning in Linear Bandits [2.372134377778215]
  We study online learning with an additional offline dataset in the linear bandit setting.<n>We propose a linear bandit algorithm that balances this tradeoff: it relies on offline data during early rounds, and increasingly favors exploration as the horizon grows.
  arXiv  Detail & Related papers  (2026-06-03T00:18:46Z)
• Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies [41.452036409068235]
  Data collection is crucial for learning robust world models in model-based reinforcement learning.<n>Online vs. offline data on world models and thus on the resulting task performance have not been thoroughly studied in the literature.<n>We identify a key challenge behind performance degradation of offline agents: encountering Out-Of-Distribution states at test time.<n>We demonstrate that this issue can be mitigated by allowing for additional online interactions in a fixed or adaptive schedule.
  arXiv  Detail & Related papers  (2025-09-06T14:52:33Z)
• Active Advantage-Aligned Online Reinforcement Learning with Offline Data [56.98480620108727]
  We introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned sampling strategy.<n>We demonstrate that our method outperforms competing online RL techniques that leverage offline data.
  arXiv  Detail & Related papers  (2025-02-11T20:31:59Z)
• Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Data [64.74333980417235]
  We show that retaining offline data is unnecessary as long as we use a properly-designed online RL approach for fine-tuning offline RL.<n>We show that Warm-start RL (WSRL) is able to fine-tune without retaining any offline data, and is able to learn faster and attains higher performance than existing algorithms.
  arXiv  Detail & Related papers  (2024-12-10T18:57:12Z)
• Understanding the performance gap between online and offline alignment algorithms [63.137832242488926]
  We show that offline algorithms train policy to become good at pairwise classification, while online algorithms are good at generations. This hints at a unique interplay between discriminative and generative capabilities, which is greatly impacted by the sampling process. Our study sheds light on the pivotal role of on-policy sampling in AI alignment, and hints at certain fundamental challenges of offline alignment algorithms.
  arXiv  Detail & Related papers  (2024-05-14T09:12:30Z)
• Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learning [66.43003402281659]
  A central question boils down to how to efficiently utilize online data collection to strengthen and complement the offline dataset. We design a three-stage hybrid RL algorithm that beats the best of both worlds -- pure offline RL and pure online RL. The proposed algorithm does not require any reward information during data collection.
  arXiv  Detail & Related papers  (2023-05-17T15:17:23Z)
• MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observations [43.9636309593499]
  We study a new paradigm for sequential decision making, called offline policy learning from observations (PLfO) We present a generic approach to offline PLfO, called $textbfM$odality-agnostic $textbfA$dversarial $textbfH$ypothesis $textbfA$daptation for $textbfL$earning from $textbfO$bservations (MAHALO)
  arXiv  Detail & Related papers  (2023-03-30T05:27:46Z)
• Adaptive Policy Learning for Offline-to-Online Reinforcement Learning [27.80266207283246]
  We consider an offline-to-online setting where the agent is first learned from the offline dataset and then trained online. We propose a framework called Adaptive Policy Learning for effectively taking advantage of offline and online data.
  arXiv  Detail & Related papers  (2023-03-14T08:13:21Z)
• Leveraging Offline Data in Online Reinforcement Learning [24.18369781999988]
  Two central paradigms have emerged in the reinforcement learning (RL) community: online RL and offline RL. In the online RL setting, the agent has no prior knowledge of the environment, and must interact with it in order to find an $epsilon$-optimal policy. In the offline RL setting, the learner instead has access to a fixed dataset to learn from, but is unable to otherwise interact with the environment, and must obtain the best policy it can from this offline data.
  arXiv  Detail & Related papers  (2022-11-09T15:39:32Z)
• Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning [80.25648265273155]
  Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.
  arXiv  Detail & Related papers  (2022-10-25T09:08:26Z)
• Online Continual Learning with Natural Distribution Shifts: An Empirical Study with Visual Data [101.6195176510611]
  "Online" continual learning enables evaluating both information retention and online learning efficacy. In online continual learning, each incoming small batch of data is first used for testing and then added to the training set, making the problem truly online. We introduce a new benchmark for online continual visual learning that exhibits large scale and natural distribution shifts.
  arXiv  Detail & Related papers  (2021-08-20T06:17:20Z)
• Representation Matters: Offline Pretraining for Sequential Decision Making [27.74988221252854]
  In this paper, we consider a slightly different approach to incorporating offline data into sequential decision-making. We find that the use of pretraining with unsupervised learning objectives can dramatically improve the performance of policy learning algorithms.
  arXiv  Detail & Related papers  (2021-02-11T02:38:12Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.