Soft $Q(λ)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces

• URL: http://arxiv.org/abs/2604.13780v1
• Date: Wed, 15 Apr 2026 12:10:45 GMT
• Title: Soft $Q(λ)$: A multi-step off-policy method for entropy regularised reinforcement learning using eligibility traces
• Abstract summary: We first present a formal $n$-step formulation for soft Q-learning and then extend this framework to the fully off-policy case by introducing a novel Soft Tree Backup operator.<n>We unify these developments into Soft $Q()$, an elegant online, off-policy, eligibility trace framework that allows for efficient credit assignment under arbitrary behaviour policies.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Soft Q-learning has emerged as a versatile model-free method for entropy-regularised reinforcement learning, optimising for returns augmented with a penalty on the divergence from a reference policy. Despite its success, the multi-step extensions of soft Q-learning remain relatively unexplored and limited to on-policy action sampling under the Boltzmann policy. In this brief research note, we first present a formal $n$-step formulation for soft Q-learning and then extend this framework to the fully off-policy case by introducing a novel Soft Tree Backup operator. Finally, we unify these developments into Soft $Q(λ)$, an elegant online, off-policy, eligibility trace framework that allows for efficient credit assignment under arbitrary behaviour policies. Our derivations propose a model-free method for learning entropy-regularised value functions that can be utilised in future empirical experiments.

Related papers

• Exact Model-Free Policy Iteration for Co-safe LTL Planning [2.5735476569508995]
  This work studies model-free reinforcement learning for co-safe linear temporal logic objectives in finite Markov decision processes.<n>We develop a new two-step model-free reinforcement learning method that first uses a discounted surrogate to identify a clamp set.<n>We prove almost-sure convergence of the policy evaluation step and finite termination of the policy iteration algorithm at an optimal policy.
  arXiv  Detail & Related papers  (2026-08-05T16:58:25Z)
• POETS: Uncertainty-Aware LLM Optimization via Compute-Efficient Policy Ensembles [35.88280776565273]
  POETS is a novel framework that bridges uncertainty quantification and policy optimization.<n>Our approach is grounded in the insight that policies trained with Kullback-Leibler (KL) regularization implicitly encode an underlying reward function.<n>We demonstrate POETS achieves state-of-the-art sample efficiency across diverse scientific discovery domains.
  arXiv  Detail & Related papers  (2026-05-08T14:16:32Z)
• ExO-PPO: an Extended Off-policy Proximal Policy Optimization Algorithm [2.6813717321945103]
  We propose a new PPO variant based on the stability guarantee from conservative on-policy iteration with a more efficient off-policy data utilization.<n>Compared with PPO and some other state-of-the-art variants, we demonstrate an improved performance of ExO-PPO with balanced sample efficiency and stability on varied tasks.
  arXiv  Detail & Related papers  (2026-02-10T12:29:57Z)
• SOUP: Token-level Single-sample Mix-policy Reinforcement Learning for Large Language Models [67.41779761651924]
  SOUP is a framework that unifies off- and on-policy learning within individual samples at the token level.<n>It consistently outperforms standard on-policy training and existing off-policy extensions.
  arXiv  Detail & Related papers  (2026-01-29T09:56:15Z)
• Q-learning with Adjoint Matching [58.78551025170267]
  We propose Q-learning with Adjoint Matching (QAM), a novel TD-based reinforcement learning (RL) algorithm.<n>QAM sidesteps two challenges by leveraging adjoint matching, a recently proposed technique in generative modeling.<n>It consistently outperforms prior approaches on hard, sparse reward tasks in both offline and offline-to-online RL.
  arXiv  Detail & Related papers  (2026-01-20T18:45:34Z)
• Deep Learning and Elicitability for McKean-Vlasov FBSDEs With Common Noise [2.421459418045937]
  We present a novel numerical method for solving McKean-Vlasov forward-backward differential equations (MV-FBSDEs) with common noise.<n>The key innovation involves elicitability to derive a path-wise loss function, enabling efficient training of neural networks to approximate both the backward process and the conditional expectations arising from common noise.<n>We validate the algorithm on a systemic risk inter-bank borrowing and lending model, where analytical solutions exist, demonstrating accurate recovery of the true solution.
  arXiv  Detail & Related papers  (2025-12-16T23:39:31Z)
• Stabilizing Policy Gradients for Sample-Efficient Reinforcement Learning in LLM Reasoning [77.92320830700797]
  Reinforcement Learning has played a central role in enabling reasoning capabilities of Large Language Models.<n>We propose a tractable computational framework that tracks and leverages curvature information during policy updates.<n>The algorithm, Curvature-Aware Policy Optimization (CAPO), identifies samples that contribute to unstable updates and masks them out.
  arXiv  Detail & Related papers  (2025-10-01T12:29:32Z)
• Relative Entropy Pathwise Policy Optimization [66.03329137921949]
  We present an on-policy algorithm that trains Q-value models purely from on-policy trajectories.<n>We show how to combine policies for exploration with constrained updates for stable training, and evaluate important architectural components that stabilize value function learning.
  arXiv  Detail & Related papers  (2025-07-15T06:24:07Z)
• Model-based Safe Deep Reinforcement Learning via a Constrained Proximal Policy Optimization Algorithm [4.128216503196621]
  We propose an On-policy Model-based Safe Deep RL algorithm in which we learn the transition dynamics of the environment in an online manner. We show that our algorithm is more sample efficient and results in lower cumulative hazard violations as compared to constrained model-free approaches.
  arXiv  Detail & Related papers  (2022-10-14T06:53:02Z)
• State Augmented Constrained Reinforcement Learning: Overcoming the Limitations of Learning with Rewards [88.30521204048551]
  A common formulation of constrained reinforcement learning involves multiple rewards that must individually accumulate to given thresholds. We show a simple example in which the desired optimal policy cannot be induced by any weighted linear combination of rewards. This work addresses this shortcoming by augmenting the state with Lagrange multipliers and reinterpreting primal-dual methods.
  arXiv  Detail & Related papers  (2021-02-23T21:07:35Z)
• Greedy Multi-step Off-Policy Reinforcement Learning [14.720255341733413]
  We propose a novel bootstrapping method, which greedily takes the maximum value among the bootstrapping values with varying steps. Experiments reveal that the proposed methods are reliable, easy to implement, and achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-02-23T14:32:20Z)
• Model-Augmented Q-learning [112.86795579978802]
  We propose a MFRL framework that is augmented with the components of model-based RL. Specifically, we propose to estimate not only the $Q$-values but also both the transition and the reward with a shared network. We show that the proposed scheme, called Model-augmented $Q$-learning (MQL), obtains a policy-invariant solution which is identical to the solution obtained by learning with true reward.
  arXiv  Detail & Related papers  (2021-02-07T17:56:50Z)
• Strictly Batch Imitation Learning by Energy-based Distribution Matching [104.33286163090179]
  Consider learning a policy purely on the basis of demonstrated behavior -- that is, with no access to reinforcement signals, no knowledge of transition dynamics, and no further interaction with the environment. One solution is simply to retrofit existing algorithms for apprenticeship learning to work in the offline setting. But such an approach leans heavily on off-policy evaluation or offline model estimation, and can be indirect and inefficient. We argue that a good solution should be able to explicitly parameterize a policy, implicitly learn from rollout dynamics, and operate in an entirely offline fashion.
  arXiv  Detail & Related papers  (2020-06-25T03:27:59Z)
• Entropy-Augmented Entropy-Regularized Reinforcement Learning and a Continuous Path from Policy Gradient to Q-Learning [5.185562073975834]
  entropy augmentation is reformulated and leads to a motivation to introduce an additional entropy term to the objective function. It results in a policy which monotonically improves while interpolating from the current policy to the softmax greedy policy.
  arXiv  Detail & Related papers  (2020-05-18T16:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.