Direct Soft-Policy Sampling via Langevin Dynamics

• URL: http://arxiv.org/abs/2602.07873v1
• Date: Sun, 08 Feb 2026 09:01:54 GMT
• Title: Direct Soft-Policy Sampling via Langevin Dynamics
• Abstract summary: Langevin Q-Learning (NCLQL) learns a noise-conditioned Q-function that induces a sequence progressively smoothed value entropy.<n>On OpenAI Gym MuJoCo benchmarks, NC-LQL provides a simple yet powerful solution for online RL.
• Score: 4.94481688445056
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Soft policies in reinforcement learning define policies as Boltzmann distributions over state-action value functions, providing a principled mechanism for balancing exploration and exploitation. However, realizing such soft policies in practice remains challenging. Existing approaches either depend on parametric policies with limited expressivity or employ diffusion-based policies whose intractable likelihoods hinder reliable entropy estimation in soft policy objectives. We address this challenge by directly realizing soft-policy sampling via Langevin dynamics driven by the action gradient of the Q-function. This perspective leads to Langevin Q-Learning (LQL), which samples actions from the target Boltzmann distribution without explicitly parameterizing the policy. However, directly applying Langevin dynamics suffers from slow mixing in high-dimensional and non-convex Q-landscapes, limiting its practical effectiveness. To overcome this, we propose Noise-Conditioned Langevin Q-Learning (NC-LQL), which integrates multi-scale noise perturbations into the value function. NC-LQL learns a noise-conditioned Q-function that induces a sequence of progressively smoothed value landscapes, enabling sampling to transition from global exploration to precise mode refinement. On OpenAI Gym MuJoCo benchmarks, NC-LQL achieves competitive performance compared to state-of-the-art diffusion-based methods, providing a simple yet powerful solution for online RL.

Related papers

• Test-Time Gradient Guidance of Flow Policies in Reinforcement Learning [50.738952715864116]
  Expressive continuous control policies form the backbone of advances in scaling imitation learning for simulated and real robot control.<n>We propose QGF (Q-Guided Flow), an RL algorithm that performs policy optimization entirely at test time.<n> Empirically, QGF outperforms prior test-time RL methods on single-task and goal-conditioned offline RL benchmarks.
  arXiv  Detail & Related papers  (2026-06-09T16:45:57Z)
• Truncated Rectified Flow Policy for Reinforcement Learning with One-Step Sampling [3.6266846456338695]
  Maximum entropy reinforcement learning (MaxEnt RL) has become a standard framework for sequential decision making.<n>We propose Truncated Rectified Flow Policy, a framework built on a hybrid deterministic-stochastic architecture.
  arXiv  Detail & Related papers  (2026-04-10T09:44:28Z)
• Flow Matching Policy with Entropy Regularization [16.47598359293598]
  Flow Matching Policy with Entropy Regularization (FMER) is an Ordinary Differential Equation (ODE)-based online RL framework.<n>FMER parameterizes the policy via flow matching and samples actions along a straight probability path, motivated by optimal transport.<n> Experiments on sparse multi-goal FrankaKitchen benchmarks demonstrate that FMER outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-03-18T13:00:20Z)
• Reparameterization Flow Policy Optimization [35.59197802340267]
  Flow policies generate actions via differentiable ODE integration.<n>RFO computes policy gradients by backpropagating jointly through the flow generation process and system dynamics.<n>RFO achieves almost $2times$ the reward of the state-of-the-art baseline.
  arXiv  Detail & Related papers  (2026-02-03T13:22:08Z)
• Stabilizing the Q-Gradient Field for Policy Smoothness in Actor-Critic [7.536387580547838]
  We argue that policy non-smoothness is governed by the differential geometry of the critic.<n>We introduce PAVE, a critic-centric regularization framework.<n>PAVE rectifies the learning signal by minimizing the Q-gradient volatility while preserving local curvature.
  arXiv  Detail & Related papers  (2026-01-30T13:32:52Z)
• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Q-STAC: Q-Guided Stein Variational Model Predictive Actor-Critic [12.837649598521102]
  This paper introduces the Q-guided STein variational model predictive Actor-Critic (Q-STAC) framework for continuous control tasks.<n>Our method optimize control sequences directly using learned Q-values as objectives, eliminating the need for explicit cost function design.<n>Experiments on 2D navigation and robotic manipulation tasks demonstrate that Q-STAC achieves superior sample efficiency, robustness, and optimality compared to state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2025-07-09T07:53:53Z)
• Sampling from Energy-based Policies using Diffusion [18.135501150108894]
  Energy-based policies offer a flexible framework for modeling complex, multimodal behaviors in reinforcement learning.<n>Existing methods typically use simpler parametric distributions, like Gaussians, for policy representation.<n>We introduce a diffusion-based approach for sampling from energy-based policies, where the negative Q-function defines the energy function.
  arXiv  Detail & Related papers  (2024-10-02T08:09:33Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Projected Off-Policy Q-Learning (POP-QL) for Stabilizing Offline Reinforcement Learning [57.83919813698673]
  Projected Off-Policy Q-Learning (POP-QL) is a novel actor-critic algorithm that simultaneously reweights off-policy samples and constrains the policy to prevent divergence and reduce value-approximation error. In our experiments, POP-QL not only shows competitive performance on standard benchmarks, but also out-performs competing methods in tasks where the data-collection policy is significantly sub-optimal.
  arXiv  Detail & Related papers  (2023-11-25T00:30:58Z)
• Boosting Continuous Control with Consistency Policy [14.78980095597872]
  We propose a novel time-efficiency method named Consistency Policy with Q-Learning (CPQL) By establishing a mapping from the reverse diffusion trajectories to the desired policy, we simultaneously address the issues of time efficiency and inaccurate guidance. CPQL achieves new state-of-the-art performance on 11 offline and 21 online tasks, significantly improving inference speed by nearly 45 times compared to Diffusion-QL.
  arXiv  Detail & Related papers  (2023-10-10T06:26:05Z)
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies [72.4573167739712]
  Implicit Q-learning (IQL) trains a Q-function using only dataset actions through a modified Bellman backup. It is unclear which policy actually attains the values represented by this trained Q-function. We introduce Implicit Q-learning (IDQL), combining our general IQL critic with the policy extraction method.
  arXiv  Detail & Related papers  (2023-04-20T18:04:09Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.