Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation

• URL: http://arxiv.org/abs/2604.13966v1
• Date: Wed, 15 Apr 2026 15:17:40 GMT
• Title: Provably Efficient Offline-to-Online Value Adaptation with General Function Approximation
• Abstract summary: We study value adaptation in offline-to-online reinforcement learning under general function approximation.<n>We propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL.
• Score: 20.941097002434944
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study value adaptation in offline-to-online reinforcement learning under general function approximation. Starting from an imperfect offline pretrained $Q$-function, the learner aims to adapt it to the target environment using only a limited amount of online interaction. We first characterize the difficulty of this setting by establishing a minimax lower bound, showing that even when the pretrained $Q$-function is close to optimal $Q^\star$, online adaptation can be no more efficient than pure online RL on certain hard instances. On the positive side, under a novel structural condition on the offline-pretrained value functions, we propose O2O-LSVI, an adaptation algorithm with problem-dependent sample complexity that provably improves over pure online RL. Finally, we complement our theory with neural-network experiments that demonstrate the practical effectiveness of the proposed method.

Related papers

• The Golden Subspace: Where Efficiency Meets Generalization in Continual Test-Time Adaptation [86.83038620308423]
  Continual Test-Time Adaptation (CTTA) aims to enable models to adapt online to unlabeled data streams under distribution shift without accessing source data.<n>We propose Guided Online Low-rank Directional adaptation (GOLD), which uses a lightweight adapter to project features onto the golden subspace and learns a compact scaling vector while the subspace is dynamically updated via AGOP.
  arXiv  Detail & Related papers  (2026-03-23T12:48:38Z)
• Sample Efficient Active Algorithms for Offline Reinforcement Learning [11.11852070175351]
  offline reinforcement learning (RL) enables policy learning from static data but often suffers from poor coverage of state-action space and distributional shift problems.<n>This paper develops a rigorous sample-complexity analysis of ActiveRL through the lens of Gaussian Process (GP) uncertainty modeling.<n>Our results reveal that ActiveRL achieves near-optimal information efficiency, that is, guided uncertainty reduction leads to accelerated value-function convergence with minimal online data.
  arXiv  Detail & Related papers  (2026-02-01T14:38:07Z)
• Zero-Shot Transferable Solution Method for Parametric Optimal Control Problems [2.365391421959969]
  This paper presents a transferable solution for optimal control problems with varying objectives using function encoder (FE) policies.<n>The proposed method learns a reusable set of neural basis functions that spans the control policy space.<n> Numerical experiments across diverse dynamics, dimensions, and cost structures show our method delivers near-optimal performance with minimal overhead.
  arXiv  Detail & Related papers  (2025-09-22T20:38:05Z)
• Preference Elicitation for Offline Reinforcement Learning [59.136381500967744]
  We propose Sim-OPRL, an offline preference-based reinforcement learning algorithm.<n>Our algorithm employs a pessimistic approach for out-of-distribution data, and an optimistic approach for acquiring informative preferences about the optimal policy.
  arXiv  Detail & Related papers  (2024-06-26T15:59:13Z)
• Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning [71.02384943570372]
  Family Offline-to-Online RL (FamO2O) is a framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark.
  arXiv  Detail & Related papers  (2023-10-27T08:30:54Z)
• Efficient Methods for Non-stationary Online Learning [63.268670895111654]
  We present efficient methods for optimizing dynamic regret and adaptive regret.<n>The proposed algorithms require only one gradient query and one function evaluation at each round.<n>We also study an even stronger measure, namely "interval dynamic regret", and reduce the number of projections per round from $O(log2 T)$ to $1$.
  arXiv  Detail & Related papers  (2023-09-16T07:30:12Z)
• Semi-Offline Reinforcement Learning for Optimized Text Generation [35.1606951874979]
  In reinforcement learning (RL), there are two major settings for interacting with the environment: online and offline. Online methods explore the environment at significant time cost, and offline methods efficiently obtain reward signals by sacrificing exploration capability. We propose semi-offline RL, a novel paradigm that smoothly transits from offline to online settings, balances exploration capability and training cost, and provides a theoretical foundation for comparing different RL settings.
  arXiv  Detail & Related papers  (2023-06-16T09:24:29Z)
• On Efficient Online Imitation Learning via Classification [17.416831207557603]
  We study classification-based online imitation learning (abbrev. $textbfCOIL$) and the fundamental feasibility to design oracle-efficient regret-minimization algorithms. Our work puts classification-based online imitation learning, an important IL setup, into a firmer foundation.
  arXiv  Detail & Related papers  (2022-09-26T17:34:36Z)
• Proximal Point Imitation Learning [48.50107891696562]
  We develop new algorithms with rigorous efficiency guarantees for infinite horizon imitation learning. We leverage classical tools from optimization, in particular, the proximal-point method (PPM) and dual smoothing. We achieve convincing empirical performance for both linear and neural network function approximation.
  arXiv  Detail & Related papers  (2022-09-22T12:40:21Z)
• Adaptivity and Non-stationarity: Problem-dependent Dynamic Regret for Online Convex Optimization [70.4342220499858]
  We introduce novel online algorithms that can exploit smoothness and replace the dependence on $T$ in dynamic regret with problem-dependent quantities. Our results are adaptive to the intrinsic difficulty of the problem, since the bounds are tighter than existing results for easy problems and safeguard the same rate in the worst case.
  arXiv  Detail & Related papers  (2021-12-29T02:42:59Z)
• Offline Neural Contextual Bandits: Pessimism, Optimization and Generalization [42.865641215856925]
  We propose a provably efficient offline contextual bandit with neural network function approximation. We show that our method generalizes over unseen contexts under a milder condition for distributional shift than the existing OPL works. We also demonstrate the empirical effectiveness of our method in a range of synthetic and real-world OPL problems.
  arXiv  Detail & Related papers  (2021-11-27T03:57:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.