Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning

• URL: http://arxiv.org/abs/2605.30776v1
• Date: Fri, 29 May 2026 03:08:42 GMT
• Title: Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
• Abstract summary: Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions.<n>Existing work aims to mitigate the harm of this shift by finetuning the policy on trajectory data sampled from a diffusion model.<n>We propose an efficient textbfDiffusion textbfUncertainty-textbfAware framework for offline-to-online reinforcement textbfL.
• Score: 16.845581305736363
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Offline-to-Online Reinforcement Learning (O2O-RL) leverages an offline, pre-trained policy to minimize costly online interactions. Although data-efficient, O2O-RL is susceptible to shifts between offline and online distributions. Existing work aims to mitigate the harm of this shift by finetuning the policy on trajectory data sampled from a diffusion model. Inspired by this line of work, we propose DUAL: an efficient \textbf{D}iffusion \textbf{U}ncertainty-\textbf{A}ware framework for offline-to-online reinforcement \textbf{L}earning. DUAL utilizes the prior knowledge of the diffusion model to distill a fast-sampling diffusion actor policy and transition model in the offline phase. DUAL also employs a Laplace approximation and distance transition-state-shift detection, thereby using uncertainty quantification to improve exploration versus exploitation in the online phase. We formally show that our actor loss with the Laplace approximation provides a proxy for a principled estimate of epistemic uncertainty. Empirically, DUAL improves the online expected return over O2O-RL baselines across multiple settings and environments.

Related papers

• Diffusing to Coordinate: Efficient Online Multi-Agent Diffusion Policies [51.24079409973799]
  Diffusion-based generative models are well-positioned to meet the needs of online Multi-Agent Reinforcement Learning (MARL)<n>We propose among the first underlineOnline off-policy underlineMARL framework using underlineDiffusion policies to orchestrate coordination.<n>Our key innovation is a relaxed policy objective that maximizes scaled joint entropy, facilitating effective exploration without relying on tractable likelihood.
  arXiv  Detail & Related papers  (2026-02-20T15:38:02Z)
• Boosting Fidelity for Pre-Trained-Diffusion-Based Low-Light Image Enhancement via Condition Refinement [63.54516423266521]
  Pre-Trained Diffusion-Based (PTDB) methods often sacrifice content fidelity to attain higher perceptual realism.<n>We propose a novel optimization strategy for conditioning in pre-trained diffusion models, enhancing fidelity while preserving realism and aesthetics.<n>Our approach is plug-and-play, seamlessly integrating into existing diffusion networks to provide more effective control.
  arXiv  Detail & Related papers  (2025-10-20T02:40:06Z)
• SAMG: Offline-to-Online Reinforcement Learning via State-Action-Conditional Offline Model Guidance [10.78460888734411]
  offline-to-online (O2O) reinforcement learning pre-trains models on offline data and refines policies through online fine-tuning.<n>We introduce State-Action-Conditional Offline Model Guidance (SAMG)<n>It freezes the pre-trained offline critic to provide compact offline understanding for each state-action sample.<n>It outperforms state-of-the-art O2O RL algorithms on the D4RL benchmark.
  arXiv  Detail & Related papers  (2024-10-24T10:35:02Z)
• Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learning [13.802860320234469]
  We introduce textbfEnergy-guided textbfDIffusion textbfSampling (EDIS) EDIS uses a diffusion model to extract prior knowledge from the offline dataset and employs energy functions to distill this knowledge for enhanced data generation in the online phase. We observe a notable 20% average improvement in empirical performance on MuJoCo, AntMaze, and Adroit environments.
  arXiv  Detail & Related papers  (2024-07-17T09:56:51Z)
• ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories [27.5648276335047]
  Training autonomous agents with sparse rewards is a long-standing problem in online reinforcement learning (RL) We propose a novel approach that leverages offline data to learn a generative diffusion model, coined as Adaptive Trajectory diffuser (ATraDiff) ATraDiff consistently achieves state-of-the-art performance across a variety of environments, with particularly pronounced improvements in complicated settings.
  arXiv  Detail & Related papers  (2024-06-06T17:58:15Z)
• Train Once, Get a Family: State-Adaptive Balances for Offline-to-Online Reinforcement Learning [71.02384943570372]
  Family Offline-to-Online RL (FamO2O) is a framework that empowers existing algorithms to determine state-adaptive improvement-constraint balances. FamO2O offers a statistically significant improvement over various existing methods, achieving state-of-the-art performance on the D4RL benchmark.
  arXiv  Detail & Related papers  (2023-10-27T08:30:54Z)
• Towards Robust Offline-to-Online Reinforcement Learning via Uncertainty and Smoothness [11.241036026084222]
  offline-to-online (O2O) RL provides a paradigm for improving an offline trained agent within limited online interactions. Most offline RL algorithms suffer from performance drops and fail to achieve stable policy improvement in O2O adaptation. We propose the Robust Offline-to-Online (RO2O) algorithm, designed to enhance offline policies through uncertainty and smoothness.
  arXiv  Detail & Related papers  (2023-09-29T04:42:50Z)
• A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning [25.123237633748193]
  offline-to-online reinforcement learning can be challenging due to constrained exploratory behavior and state-action distribution shift. We propose a Simple Unified uNcertainty-Guided (SUNG) framework, which unifies the solution to both challenges with the tool of uncertainty. SUNG achieves state-of-the-art online finetuning performance when combined with different offline RL methods.
  arXiv  Detail & Related papers  (2023-06-13T05:22:26Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)
• Efficient Diffusion Policies for Offline Reinforcement Learning [85.73757789282212]
  Diffsuion-QL significantly boosts the performance of offline RL by representing a policy with a diffusion model. We propose efficient diffusion policy (EDP) to overcome these two challenges. EDP constructs actions from corrupted ones at training to avoid running the sampling chain.
  arXiv  Detail & Related papers  (2023-05-31T17:55:21Z)
• Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning [80.25648265273155]
  Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.
  arXiv  Detail & Related papers  (2022-10-25T09:08:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.