Drift Q-Learning

• URL: http://arxiv.org/abs/2606.00350v1
• Date: Fri, 29 May 2026 20:42:30 GMT
• Title: Drift Q-Learning
• Abstract summary: We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement.<n>The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data.<n>On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods.
• Score: 18.076463313217175
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Offline reinforcement learning requires improving a policy from fixed data while avoiding out-of-distribution actions with unreliable value estimates. Diffusion and flow policies handle this trade-off by modeling the behavior distribution to regularize the RL objective, but they require iterative denoising, solver integrations, and in more efficient variants, distillation or other approximations at inference. We propose DriftQL, which combines a drift-based behavioral regularizer with critic-driven policy improvement. The value signal biases the policy toward high-value regions of the data support, while attraction and repulsion together keep generated actions near the data and prevent collapse onto a single mode. DriftQL is implemented as a single network with a unified training objective and generates actions in a single forward pass. On D4RL and OGBench, DriftQL consistently outperforms diffusion and flow methods, advancing the state of the art. Under degraded data quality, where the baselines visibly struggle, DriftQL remains close to its clean-data performance, positioning it as a promising alternative to diffusion and flow-based methods while maintaining the simplicity and efficiency of deterministic approaches. Project page: https://driftql.github.io/

Related papers

• Trust the Batch, On- or Off-Policy: Adaptive Policy Optimization for RL Post-Training [50.86545293331458]
  Reinforcement learning is structurally harder than supervised learning because the policy changes the data distribution it learns from.<n>We present a simple yet effective batch-adaptive objective that replaces fixed clipping with the normalized effective sample size of the policy ratios.
  arXiv  Detail & Related papers  (2026-05-12T16:44:47Z)
• ScoRe-Flow: Complete Distributional Control via Score-Based Reinforcement Learning for Flow Matching [14.421249343302028]
  Flow Matching (FM) policies have emerged as an efficient backbone for robotic control.<n>We propose ScoRe-Flow, a score-based RL fine-tuning method to achieve decoupled control over the mean and variance of transitions.
  arXiv  Detail & Related papers  (2026-04-13T03:56:37Z)
• Amortizing Trajectory Diffusion with Keyed Drift Fields [2.5352713493505785]
  Keyed Drifting Policies is a one-step trajectory generator trained with a drift-field objective.<n>It repels generated trajectories toward condition-matched dataset windows.<n>It achieves strong performance with one-step inference and substantially lower planning latency than diffusion sampling.
  arXiv  Detail & Related papers  (2026-03-14T17:56:15Z)
• Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learning [10.037416068775853]
  We introduce Guided Flow Policy, which couples a multi-step flow-matching policy with a distilled one-step actor.<n>The actor directs the flow policy through weighted behavior cloning to focus on cloning high-value actions from the dataset.<n>This mutual guidance enables GFP to achieve state-of-the-art performance across 144 state and pixel-based tasks.
  arXiv  Detail & Related papers  (2025-12-03T17:05:58Z)
• Steering Vision-Language-Action Models as Anti-Exploration: A Test-Time Scaling Approach [78.4812458793128]
  We propose textbfTACO, a test-time-scaling framework that applies a lightweight pseudo-count estimator as a high-fidelity verifier of action chunks.<n>Our method resembles the classical anti-exploration principle in offline reinforcement learning (RL), and being gradient-free, it incurs significant computational benefits.
  arXiv  Detail & Related papers  (2025-12-02T14:42:54Z)
• ResAD: Normalized Residual Trajectory Modeling for End-to-End Autonomous Driving [64.42138266293202]
  ResAD is a Normalized Residual Trajectory Modeling framework.<n>It reframes the learning task to predict the residual deviation from an inertial reference.<n>On the NAVSIM benchmark, ResAD achieves a state-of-the-art PDMS of 88.6 using a vanilla diffusion policy.
  arXiv  Detail & Related papers  (2025-10-09T17:59:36Z)
• FedDAA: Dynamic Client Clustering for Concept Drift Adaptation in Federated Learning [6.327777127276663]
  We propose FedDAA, a dynamic clustered FL framework designed to adapt to multi-source concept drift while preserving valuable historical knowledge.<n>Specifically, FedDAA integrates three modules: a cluster number determination module to find the optimal number of clusters; a real drift detection module to distinguish real drift from virtual/label drift; and a concept drift adaptation module to adapt to new data while retaining useful historical information.<n>Experiments show that FedDAA achieves 7.84% to 8.52% accuracy improvements over state-of-the-art methods on Fashion-MNIST, CIFAR-10, and CIFAR-100.
  arXiv  Detail & Related papers  (2025-06-26T07:09:08Z)
• FlowTS: Time Series Generation via Rectified Flow [67.41208519939626]
  FlowTS is an ODE-based model that leverages rectified flow with straight-line transport in probability space.<n>For unconditional setting, FlowTS achieves state-of-the-art performance, with context FID scores of 0.019 and 0.011 on Stock and ETTh datasets.<n>For conditional setting, we have achieved superior performance in solar forecasting.
  arXiv  Detail & Related papers  (2024-11-12T03:03:23Z)
• Diffusion Policies creating a Trust Region for Offline Reinforcement Learning [66.17291150498276]
  We introduce a dual policy approach, Diffusion Trusted Q-Learning (DTQL), which comprises a diffusion policy for pure behavior cloning and a practical one-step policy. DTQL eliminates the need for iterative denoising sampling during both training and inference, making it remarkably computationally efficient. We show that DTQL could not only outperform other methods on the majority of the D4RL benchmark tasks but also demonstrate efficiency in training and inference speeds.
  arXiv  Detail & Related papers  (2024-05-30T05:04:33Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.