Reinforcement Learning via Value Gradient Flow

• URL: http://arxiv.org/abs/2604.14265v1
• Date: Wed, 15 Apr 2026 17:12:56 GMT
• Title: Reinforcement Learning via Value Gradient Flow
• Abstract summary: We study behavior-regularized reinforcement learning (RL), where regularization toward a reference distribution is essential to prevent value over-optimization.<n>We propose Value Gradient Flow (VGF), a scalable new paradigm for behavior-regularized RL.<n>VGF casts behavior-regularized RL as an optimal transport problem that maps the reference distribution to the value-induced optimal policy distribution.
• Score: 32.59594977900539
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study behavior-regularized reinforcement learning (RL), where regularization toward a reference distribution (the dataset in offline RL or the base model in LLM RL finetuning) is essential to prevent value over-optimization caused by erroneous out-of-distribution extrapolation. Existing methods either rely on reparameterized policy gradient, which are difficult to scale to large generative models, or on reject sampling, which can be overly conservative when attempting to move beyond the behavior support. In this paper, we propose Value Gradient Flow (VGF), a scalable new paradigm for behavior-regularized RL. VGF casts behavior-regularized RL as an optimal transport problem that maps the reference distribution to the value-induced optimal policy distribution. We solve this transport problem via discrete gradient flow, where value gradients guide particles initialized from the reference distribution. Our analysis shows that VGF imposes regularization implicitly by controlling the transport budget. VGF eliminates explicit policy parameterization while remaining expressive and flexible, this enables adaptive test-time scaling by adjusting the transport budget. Extensive experiments demonstrate that VGF significantly outperforms prior methods, achieving state-of-the-art results on offline RL benchmarks (D4RL, OGBench) and LLM RL tasks. Code and runs can be found at https://ryanxhr.github.io/vgf.

Related papers

• Value-Gradient Hypothesis of RL for LLMs [1.983259889617852]
  Reinforcement learning substantially improves pretrained language models.<n>It remains understudied why critic-free methods such as PPO and GRPO work as well as they do, and when they should provide the largest gains.
  arXiv  Detail & Related papers  (2026-05-20T19:09:05Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• Controllable Exploration in Hybrid-Policy RLVR for Multi-Modal Reasoning [88.42566960813438]
  CalibRL is a hybrid-policy RLVR framework that supports controllable exploration with expert guidance.<n>CalibRL increases policy entropy in a guided manner and clarifies the target distribution.<n>Experiments across eight benchmarks, including both in-domain and out-of-domain settings, demonstrate consistent improvements.
  arXiv  Detail & Related papers  (2026-02-22T07:23:36Z)
• Zero-Order Optimization for LLM Fine-Tuning via Learnable Direction Sampling [40.94400211806987]
  We propose a policy-driven ZO framework that treats the sampling distribution over perturbation directions as a learnable policy.<n>We show that learned sampling improves quality gradient information and relax the explicit dependence on $d$ in convergence bounds.<n>Our results suggest that adaptive direction sampling is a promising route to make ZO fine-tuning viable at scale.
  arXiv  Detail & Related papers  (2026-02-14T08:01:41Z)
• Toward Adaptive Grid Resilience: A Gradient-Free Meta-RL Framework for Critical Load Restoration [6.573424608566227]
  Reinforcement learning can optimize sequential decisions under uncertainty.<n>Standard RL often generalizes poorly and requires extensive retraining for new outage configurations or generation patterns.<n>We propose a meta-guided gradient-free RL framework that learns a transferable initialization from historical outage experiences.
  arXiv  Detail & Related papers  (2026-01-16T03:36:07Z)
• Boundary-Guided Policy Optimization for Memory-efficient RL of Diffusion Large Language Models [53.339700196282905]
  A key challenge in applying reinforcement learning to large language models (dLLMs) is the intractability of their likelihood functions.<n>We propose a memory-efficient RL algorithm that maximizes a specially constructed lower bound of the ELBO-based objective.<n> Experiments show that BGPO significantly outperforms previous RL algorithms for dLLMs in math problem solving, code generation, and planning tasks.
  arXiv  Detail & Related papers  (2025-10-13T17:47:50Z)
• Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Models [29.090093552573766]
  We propose an offline RL post-training objective for Vision-Language-Action (VLA) flow models.<n>We then induce an efficient and feasible offline RL fine-tuning algorithm -- Adaptive Reinforced Flow Matching (ARFM)<n>ARFM exhibits excellent generalization, robustness, few-shot learning, and continuous learning performance.
  arXiv  Detail & Related papers  (2025-09-04T09:48:43Z)
• Relative Trajectory Balance is equivalent to Trust-PCL [72.58731629381032]
  Relative Trajectory Balance (RTB) aims to improve fine-tuning in sequential generative models.<n>This paper establishes an equivalence between RTB and Trust-PCL, an off-policy RL method with KL regularization.
  arXiv  Detail & Related papers  (2025-09-01T17:17:25Z)
• Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model [57.20064815347607]
  offline reinforcement learning (RL) recently gains growing interests from RL researchers.<n>The performance of offline RL suffers from the out-of-distribution problem, which can be corrected by feedback in online RL.<n>In this paper, we first build a bridge over the batch data and the performance of offline RL algorithms theoretically.<n>We show that in task-agnostic settings, a series of policies trained by unsupervised RL can minimize the worst-case regret in the performance gap.
  arXiv  Detail & Related papers  (2025-06-24T14:08:36Z)
• Understanding Reinforcement Learning-Based Fine-Tuning of Diffusion Models: A Tutorial and Review [63.31328039424469]
  This tutorial provides a comprehensive survey of methods for fine-tuning diffusion models to optimize downstream reward functions. We explain the application of various RL algorithms, including PPO, differentiable optimization, reward-weighted MLE, value-weighted sampling, and path consistency learning.
  arXiv  Detail & Related papers  (2024-07-18T17:35:32Z)
• Generative Modeling with Flow-Guided Density Ratio Learning [12.192867460641835]
  Flow-Guided Density Ratio Learning (FDRL) is a simple and scalable approach to generative modeling. We show that FDRL can generate images of dimensions as high as $128times128$, as well as outperform existing gradient flow baselines on quantitative benchmarks.
  arXiv  Detail & Related papers  (2023-03-07T07:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.