Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms

• URL: http://arxiv.org/abs/2604.27378v1
• Date: Thu, 30 Apr 2026 03:41:00 GMT
• Title: Continuous-time q-learning for mean-field control with common noise, part-II: q-learning algorithms
• Abstract summary: We further devise q-learning algorithms for mean-field control with controlled common noise.<n>We propose the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the improved Iq-function.<n>In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.
• Score: 7.003100820506273
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper is a continuation work of Ren et al. (2026) aiming to further devise q-learning algorithms for mean-field control (MFC) with controlled common noise. Based on the relaxed control formulation, we first establish the martingale condition of the value function and the Iq-function by evaluating along the conditional state distributions generated by all test policies. As the data in the relaxed control formulation are not observable in practice, we quantify the error incurred when they are replaced by the observable ones in the exploratory formulation under discretely sampled actions. This, together with a two-layer fixed point characterization of an optimal policy in Ren et al. (2026), allows us to propose several algorithms including the Actor-Critic q-learning algorithm, in which the policy is updated in the Actor-step based on the iteration rule induced by the improved Iq-function, and the value function and Iq-function are updated in the Critic-step based on the martingale orthogonality condition using the data from the exploratory formulation. We also establish the convergence of the inner iterations in the Actor-step in an infinite-horizon linear quadratic (LQ) framework. In two examples, within and beyond LQ framework, our q-learning algorithms are implemented with satisfactory performance.

Related papers

• Continuous-time q-learning for mean-field control with common noise, part-I: Theoretical foundations [7.003100820506273]
  We investigate the continuous-time counterpart of the Q-function for entropy-regularized mean-field control (MFC) with controlled common noise.<n>We derive the exploratory Hamilton-Jacobi-Bellman (HJB) equation, in which the controlled common noise gives rise to an additional nonlinear functional of policy.<n>In the mean-field setting, we introduce the integrated q-function (Iq-function) defined on the state distribution and the policy, and it is shown that an optimal policy is identified as a two-layer fixed point to the argmax operator of the Iq-function
  arXiv  Detail & Related papers  (2026-04-30T03:37:55Z)
• Convergence of regularized agent-state-based Q-learning in POMDPs [24.164262011028246]
  We investigate the simplest form of Q-learning algorithms which we call regularized agent-state-based Q-learning (RA)<n>We show that it converges under mild technical conditions.<n>We show that a similar analysis continues to work for a variant of RA policies that learns periodic behavior.
  arXiv  Detail & Related papers  (2025-08-29T02:45:28Z)
• Continuous-time q-learning for mean-field control problems [4.3715546759412325]
  We study the q-learning, recently coined as the continuous time counterpart of Q-learning by Jia and Zhou (2023), for continuous time Mckean-Vlasov control problems. We show that two q-functions are related via an integral representation under all test policies. Based on the weak martingale condition and our proposed searching method of test policies, some model-free learning algorithms are devised.
  arXiv  Detail & Related papers  (2023-06-28T13:43:46Z)
• Hierarchical Phrase-based Sequence-to-Sequence Learning [94.10257313923478]
  We describe a neural transducer that maintains the flexibility of standard sequence-to-sequence (seq2seq) models while incorporating hierarchical phrases as a source of inductive bias during training and as explicit constraints during inference. Our approach trains two models: a discriminative derivation based on a bracketing grammar whose tree hierarchically aligns source and target phrases, and a neural seq2seq model that learns to translate the aligned phrases one-by-one.
  arXiv  Detail & Related papers  (2022-11-15T05:22:40Z)
• Maximum-Likelihood Inverse Reinforcement Learning with Finite-Time Guarantees [56.848265937921354]
  Inverse reinforcement learning (IRL) aims to recover the reward function and the associated optimal policy. Many algorithms for IRL have an inherently nested structure. We develop a novel single-loop algorithm for IRL that does not compromise reward estimation accuracy.
  arXiv  Detail & Related papers  (2022-10-04T17:13:45Z)
• Actor-Critic based Improper Reinforcement Learning [61.430513757337486]
  We consider an improper reinforcement learning setting where a learner is given $M$ base controllers for an unknown Markov decision process. We propose two algorithms: (1) a Policy Gradient-based approach; and (2) an algorithm that can switch between a simple Actor-Critic scheme and a Natural Actor-Critic scheme.
  arXiv  Detail & Related papers  (2022-07-19T05:55:02Z)
• q-Learning in Continuous Time [11.694169299062597]
  We study the continuous-time counterpart of Q-learning for reinforcement learning (RL) under the entropy-regularized, exploratory diffusion process formulation.<n>We develop a q-learning" theory around the q-function that is independent of time discretization.<n>We devise different actor-critic algorithms for solving underlying problems.
  arXiv  Detail & Related papers  (2022-07-02T02:20:41Z)
• Policy Gradient and Actor-Critic Learning in Continuous Time and Space: Theory and Algorithms [1.776746672434207]
  We study policy gradient (PG) for reinforcement learning in continuous time and space. We propose two types of the actor-critic algorithms for RL, where we learn and update value functions and policies simultaneously and alternatingly.
  arXiv  Detail & Related papers  (2021-11-22T14:27:04Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)
• Logistic Q-Learning [87.00813469969167]
  We propose a new reinforcement learning algorithm derived from a regularized linear-programming formulation of optimal control in MDPs. The main feature of our algorithm is a convex loss function for policy evaluation that serves as a theoretically sound alternative to the widely used squared Bellman error.
  arXiv  Detail & Related papers  (2020-10-21T17:14:31Z)
• SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learning [102.78958681141577]
  We present SUNRISE, a simple unified ensemble method, which is compatible with various off-policy deep reinforcement learning algorithms. SUNRISE integrates two key ingredients: (a) ensemble-based weighted Bellman backups, which re-weight target Q-values based on uncertainty estimates from a Q-ensemble, and (b) an inference method that selects actions using the highest upper-confidence bounds for efficient exploration.
  arXiv  Detail & Related papers  (2020-07-09T17:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.