Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning

• URL: http://arxiv.org/abs/2605.01663v1
• Date: Sun, 03 May 2026 01:32:11 GMT
• Title: Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
• Abstract summary: We propose Flow-Anchored Noise-conditioned Q-Learning (FAN)<n>FAN is a highly efficient and high-performing offline reinforcement learning algorithm.<n> Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance.
• Score: 4.215835343017977
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose Flow-Anchored Noise-conditioned Q-Learning (FAN), a highly efficient and high-performing offline reinforcement learning (RL) algorithm. Recent work has shown that expressive flow policies and distributional critics improve offline RL performance, but at a high computational cost. Specifically, flow policies require iterative sampling to produce a single action, and distributional critics require computation over multiple samples (e.g., quantiles) to estimate value. To address these inefficiencies while maintaining high performance, we introduce FAN. Our method employs a behavior regularization technique that utilizes only a single flow policy iteration and requires only a single Gaussian noise sample for distributional critics. Our theoretical analysis of convergence and performance bounds demonstrates that these simplifications not only improve efficiency but also lead to superior task performance. Experiments on robotic manipulation and locomotion tasks demonstrate that FAN achieves state-of-the-art performance while significantly reducing both training and inference runtimes. We release our code at https://github.com/brianlsy98/FAN.

Related papers

• Reversal Q-Learning [57.43956630070019]
  Reversal Q-learning is an off-policy RL algorithm that trains a flow policy based on prior data.<n>We show that RQL leads to the best average offline RL performance compared to state-of-the-art flow-based offline RL algorithms.
  arXiv  Detail & Related papers  (2026-06-16T05:56:10Z)
• QPILOTS: Efficient Test-Time Q-Steering for Flow Policies [20.217020870532686]
  QPILOTS is a method that leaves the original policy unmodified and steers the denoising process at inference time.<n>On a standard offline-to-online RL benchmark, QPILOTS achieves the best aggregate performance, reaching an average success rate of 90% across 50 tasks.
  arXiv  Detail & Related papers  (2026-06-11T18:22:03Z)
• FASTER: Value-Guided Sampling for Fast RL [103.55398181003262]
  FASTER is a method for getting the benefits of sampling-based test-time scaling of diffusion-based policies without the computational cost.<n>FASTER achieves the same performance while substantially reducing training and inference compute requirements.
  arXiv  Detail & Related papers  (2026-04-21T17:52:17Z)
• Beyond Normalization: Rethinking the Partition Function as a Difficulty Scheduler for RLVR [23.417956258945427]
  We propose a post-training framework that leverages accuracy estimates to prioritize informative question prompts during training.<n>PACED-RL is a promising direction for a more sample efficient distribution-matching training for LLMs.
  arXiv  Detail & Related papers  (2026-02-13T06:04:14Z)
• Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learning [14.799267729619428]
  In streaming Reinforcement Learning (RL), transitions are observed and discarded immediately after a single update.<n>We propose extending Self-Predictive Representations (SPR) to the streaming pipeline to maximize the utility of every observed frame.<n>We show that our method learns significantly richer representations, bridging the performance gap caused by the absence of a replay buffer.
  arXiv  Detail & Related papers  (2026-02-10T04:06:32Z)
• CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning [57.24524263804788]
  Code verifiers play a critical role in post-verification for LLM-based code generation.<n>Existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency.<n>We show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples.
  arXiv  Detail & Related papers  (2026-01-30T10:33:29Z)
• EfficientFlow: Efficient Equivariant Flow Policy Learning for Embodied AI [8.90880897951322]
  Generative modeling has recently shown remarkable promise for visuomotor policy learning.<n>We introduce EfficientFlow, a unified framework for efficient embodied AI with flow-based policy learning.
  arXiv  Detail & Related papers  (2025-12-01T18:59:59Z)
• One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlow [56.13949180229929]
  We introduce a one-step generative policy for offline reinforcement learning that maps noise directly to actions via a residual reformulation of MeanFlow.<n>Our method achieves strong performance in both offline and offline-to-online reinforcement learning settings.
  arXiv  Detail & Related papers  (2025-11-17T06:34:17Z)
• Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models [29.090093552573766]
  We propose an offline RL post-training objective for Vision-Language-Action (VLA) flow models.<n>We then induce an efficient and feasible offline RL fine-tuning algorithm -- Adaptive Reinforced Flow Matching (ARFM)<n>ARFM exhibits excellent generalization, robustness, few-shot learning, and continuous learning performance.
  arXiv  Detail & Related papers  (2025-09-04T09:48:43Z)
• One-Step Flow Policy Mirror Descent [52.31612487608593]
  Flow Policy Mirror Descent (FPMD) is an online RL algorithm that enables 1-step sampling during flow policy inference.<n>Our approach exploits a theoretical connection between the distribution variance and the discretization error of single-step sampling in straight flow matching models.
  arXiv  Detail & Related papers  (2025-07-31T15:51:10Z)
• Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows [53.31856123113228]
  This paper proposes Language Rectified Flow (ours) Our method is based on the reformulation of the standard probabilistic flow models. Experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
  arXiv  Detail & Related papers  (2024-03-25T17:58:22Z)
• Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods [133.85604983925282]
  We propose Simultaneous Double Q-learning with Conservative Advantage Learning (SDQ-CAL) Our algorithm realizes less biased value estimation and achieves state-of-the-art performance in a range of continuous control benchmark tasks.
  arXiv  Detail & Related papers  (2022-05-08T09:17:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.