Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning

• URL: http://arxiv.org/abs/2602.18117v1
• Date: Fri, 20 Feb 2026 10:14:00 GMT
• Title: Flow Matching with Injected Noise for Offline-to-Online Reinforcement Learning
• Authors: Yongjae Shin, Jongseong Chae, Jongeui Park, Youngchul Sung,
• Abstract summary: We propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO)<n>FINO is a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL.<n>Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.
• Score: 18.9517981804953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative models have recently demonstrated remarkable success across diverse domains, motivating their adoption as expressive policies in reinforcement learning (RL). While they have shown strong performance in offline RL, particularly where the target distribution is well defined, their extension to online fine-tuning has largely been treated as a direct continuation of offline pre-training, leaving key challenges unaddressed. In this paper, we propose Flow Matching with Injected Noise for Offline-to-Online RL (FINO), a novel method that leverages flow matching-based policies to enhance sample efficiency for offline-to-online RL. FINO facilitates effective exploration by injecting noise into policy training, thereby encouraging a broader range of actions beyond those observed in the offline dataset. In addition to exploration-enhanced flow policy training, we combine an entropy-guided sampling mechanism to balance exploration and exploitation, allowing the policy to adapt its behavior throughout online fine-tuning. Experiments across diverse, challenging tasks demonstrate that FINO consistently achieves superior performance under limited online budgets.

Related papers

• From Static to Dynamic: Enhancing Offline-to-Online Reinforcement Learning via Energy-Guided Diffusion Stratification [3.2883573376133555]
  StratDiff is a diffusion model to learn prior knowledge from the offline dataset.<n>It refines this knowledge through energy-based functions to improve policy imitation and generate offline-like actions during online fine-tuning.<n>Offline-like samples are updated using offline objectives, while online-like samples follow online learning strategies.
  arXiv  Detail & Related papers  (2025-11-05T19:48:46Z)
• Reinforcement Learning with Action Chunking [56.66655947239018]
  We present Q-chunking, a recipe for improving reinforcement learning algorithms for long-horizon, sparse-reward tasks.<n>Our recipe is designed for the offline-to-online RL setting, where the goal is to leverage an offline prior dataset to maximize the sample-efficiency of online learning.<n>Our experimental results demonstrate that Q-chunking exhibits strong offline performance and online sample efficiency, outperforming prior best offline-to-online methods on a range of long-horizon, sparse-reward manipulation tasks.
  arXiv  Detail & Related papers  (2025-07-10T17:48:03Z)
• Flow-Based Single-Step Completion for Efficient and Expressive Policy Learning [0.0]
  We propose a generative policy trained with an augmented flow-matching objective to predict direct completion vectors from intermediate flow samples.<n>Our method scales effectively to offline, offline-to-online, and online RL settings, offering substantial gains in speed and adaptability.<n>We extend SSCP to goal-conditioned RL, enabling flat policies to exploit subgoal structures without explicit hierarchical inference.
  arXiv  Detail & Related papers  (2025-06-26T16:09:53Z)
• Active Advantage-Aligned Online Reinforcement Learning with Offline Data [56.98480620108727]
  We introduce A3RL, which incorporates a novel confidence aware Active Advantage Aligned sampling strategy.<n>We demonstrate that our method outperforms competing online RL techniques that leverage offline data.
  arXiv  Detail & Related papers  (2025-02-11T20:31:59Z)
• Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimization [55.97310586039358]
  Diffusion models have garnered widespread attention in Reinforcement Learning (RL) for their powerful expressiveness and multimodality.<n>We propose a novel model-free diffusion-based online RL algorithm, Q-weighted Variational Policy Optimization (QVPO)<n>Specifically, we introduce the Q-weighted variational loss, which can be proved to be a tight lower bound of the policy objective in online RL under certain conditions.<n>We also develop an efficient behavior policy to enhance sample efficiency by reducing the variance of the diffusion policy during online interactions.
  arXiv  Detail & Related papers  (2024-05-25T10:45:46Z)
• Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning [9.341618348621662]
  We aim to find the best-performing policy within a limited budget of online interactions. We first study the major online RL exploration methods based on intrinsic rewards and UCB. We then introduce an algorithm for planning to go out-of-distribution that avoids these issues.
  arXiv  Detail & Related papers  (2023-10-09T13:47:05Z)
• Improving Offline-to-Online Reinforcement Learning with Q Conditioned State Entropy Exploration [29.891468119032]
  We study how to fine-tune offline reinforcement learning (RL) pre-trained policy. We propose Q conditioned state entropy (QCSE) as intrinsic reward. We observe significant improvements with QCSE (about 13% for CQL and 8% for Cal-QL)
  arXiv  Detail & Related papers  (2023-10-07T00:02:05Z)
• A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning [25.123237633748193]
  offline-to-online reinforcement learning can be challenging due to constrained exploratory behavior and state-action distribution shift. We propose a Simple Unified uNcertainty-Guided (SUNG) framework, which unifies the solution to both challenges with the tool of uncertainty. SUNG achieves state-of-the-art online finetuning performance when combined with different offline RL methods.
  arXiv  Detail & Related papers  (2023-06-13T05:22:26Z)
• ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles [52.34951901588738]
  We propose a novel framework called ENsemble-based Offline-To-Online (ENOTO) RL. By increasing the number of Q-networks, we seamlessly bridge offline pre-training and online fine-tuning without degrading performance. Experimental results demonstrate that ENOTO can substantially improve the training stability, learning efficiency, and final performance of existing offline RL methods.
  arXiv  Detail & Related papers  (2023-06-12T05:10:10Z)
• Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning [80.25648265273155]
  Offline reinforcement learning, by learning from a fixed dataset, makes it possible to learn agent behaviors without interacting with the environment. During online fine-tuning, the performance of the pre-trained agent may collapse quickly due to the sudden distribution shift from offline to online data. We propose to adaptively weigh the behavior cloning loss during online fine-tuning based on the agent's performance and training stability. Experiments show that the proposed method yields state-of-the-art offline-to-online reinforcement learning performance on the popular D4RL benchmark.
  arXiv  Detail & Related papers  (2022-10-25T09:08:26Z)
• MOORe: Model-based Offline-to-Online Reinforcement Learning [26.10368749930102]
  We propose a model-based Offline-to-Online Reinforcement learning (MOORe) algorithm. Experiment results show that our algorithm smoothly transfers from offline to online stages while enabling sample-efficient online adaption.
  arXiv  Detail & Related papers  (2022-01-25T03:14:57Z)
• OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning [107.6943868812716]
  In many practical applications, the situation is reversed: an agent may have access to large amounts of undirected offline experience data, while access to the online environment is severely limited. Our main insight is that, when presented with offline data composed of a variety of behaviors, an effective way to leverage this data is to extract a continuous space of recurring and temporally extended primitive behaviors. In addition to benefiting offline policy optimization, we show that performing offline primitive learning in this way can also be leveraged for improving few-shot imitation learning.
  arXiv  Detail & Related papers  (2020-10-26T14:31:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.