Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2602.17062v1
• Date: Thu, 19 Feb 2026 04:07:55 GMT
• Title: Retaining Suboptimal Actions to Follow Shifting Optima in Multi-Agent Reinforcement Learning
• Authors: Yonghyeon Jo, Sunwoo Lee, Seungyul Han,
• Abstract summary: Successive Sub-value Q-learning (S2Q) learns multiple sub-value functions to retain alternative high-value actions.<n> Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms.
• Score: 7.725456898964979
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Value decomposition is a core approach for cooperative multi-agent reinforcement learning (MARL). However, existing methods still rely on a single optimal action and struggle to adapt when the underlying value function shifts during training, often converging to suboptimal policies. To address this limitation, we propose Successive Sub-value Q-learning (S2Q), which learns multiple sub-value functions to retain alternative high-value actions. Incorporating these sub-value functions into a Softmax-based behavior policy, S2Q encourages persistent exploration and enables $Q^{\text{tot}}$ to adjust quickly to the changing optima. Experiments on challenging MARL benchmarks confirm that S2Q consistently outperforms various MARL algorithms, demonstrating improved adaptability and overall performance. Our code is available at https://github.com/hyeon1996/S2Q.

Related papers

• MARS: Harmonizing Multimodal Convergence via Adaptive Rank Search [12.345218777941108]
  Fine-tuning Multimodal Large Language Models (MLLMs) with parameter-efficient methods like Low-Rank Adaptation (LoRA) is crucial for task adaptation.<n>We introduce MARS (Multimodal Adaptive Rank Search), an approach to discover optimal rank pairs that balance training dynamics while maximizing performance.<n>Our key innovation, a proposed framework of dual scaling laws, enables this search: one law models module-specific convergence time to prune the search space to candidates with aligned dynamics, while the other predicts final task performance to select the optimal pair from the pruned set.
  arXiv  Detail & Related papers  (2026-02-28T15:58:28Z)
• AceGRPO: Adaptive Curriculum Enhanced Group Relative Policy Optimization for Autonomous Machine Learning Engineering [52.67783579040657]
  AceGRPO is a machine learning system that prioritizes tasks at the agent's learning frontier to maximize learning efficiency.<n>Our trained Ace-30B model achieves a 100% valid submission rate on MLE-Bench-Lite, approaches the performance of proprietary frontier models, and outperforms larger open-source baselines.
  arXiv  Detail & Related papers  (2026-02-08T10:55:03Z)
• Improving monotonic optimization in heterogeneous multi-agent reinforcement learning with optimal marginal deterministic policy gradient [18.64288030584699]
  heterogeneous multi-agent reinforcement learning (MARL)<n>Objectively replace the sequentially computed $Q_psi*(s,a_1:i)$ with the Optimal Marginal Q function $phi_psi*(s,a_1:i)$ derived from Q-functions.<n>Generalized Q Critic (GQC) as the critic function, employing pessimistic uncertainty-constrained loss to optimize different Q-value estimations.
  arXiv  Detail & Related papers  (2025-07-14T07:16:01Z)
• Addressing the issue of stochastic environments and local decision-making in multi-objective reinforcement learning [0.0]
  Multi-objective reinforcement learning (MORL) is a relatively new field which builds on conventional Reinforcement Learning (RL) This thesis focuses on what factors influence the frequency with which value-based MORL Q-learning algorithms learn the optimal policy for an environment.
  arXiv  Detail & Related papers  (2022-11-16T04:56:42Z)
• MA2QL: A Minimalist Approach to Fully Decentralized Multi-Agent Reinforcement Learning [63.46052494151171]
  We propose textitmulti-agent alternate Q-learning (MA2QL), where agents take turns to update their Q-functions by Q-learning. We prove that when each agent guarantees a $varepsilon$-convergence at each turn, their joint policy converges to a Nash equilibrium. Results show MA2QL consistently outperforms IQL, which verifies the effectiveness of MA2QL, despite such minimal changes.
  arXiv  Detail & Related papers  (2022-09-17T04:54:32Z)
• Simultaneous Double Q-learning with Conservative Advantage Learning for Actor-Critic Methods [133.85604983925282]
  We propose Simultaneous Double Q-learning with Conservative Advantage Learning (SDQ-CAL) Our algorithm realizes less biased value estimation and achieves state-of-the-art performance in a range of continuous control benchmark tasks.
  arXiv  Detail & Related papers  (2022-05-08T09:17:16Z)
• Q-Learning Lagrange Policies for Multi-Action Restless Bandits [35.022322303796216]
  Multi-action restless multi-armed bandits (RMABs) are a powerful framework for constrained resource allocation in which $N$ independent processes are managed. We design the first algorithms for learning good policies for Multi-action RMABs online using combinations of Lagrangian relaxation and Q-learning.
  arXiv  Detail & Related papers  (2021-06-22T19:20:09Z)
• Decomposed Soft Actor-Critic Method for Cooperative Multi-Agent Reinforcement Learning [10.64928897082273]
  Experimental results demonstrate that mSAC significantly outperforms policy-based approach COMA. In addition, mSAC achieves pretty good results on large action space tasks, such as 2c_vs_64zg and MMM2.
  arXiv  Detail & Related papers  (2021-04-14T07:02:40Z)
• Softmax with Regularization: Better Value Estimation in Multi-Agent Reinforcement Learning [72.28520951105207]
  Overestimation in $Q$-learning is an important problem that has been extensively studied in single-agent reinforcement learning. We propose a novel regularization-based update scheme that penalizes large joint action-values deviating from a baseline. We show that our method provides a consistent performance improvement on a set of challenging StarCraft II micromanagement tasks.
  arXiv  Detail & Related papers  (2021-03-22T14:18:39Z)
• Meta-Learning with Neural Tangent Kernels [58.06951624702086]
  We propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK) Within this paradigm, we introduce two meta-learning algorithms, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework. We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory.
  arXiv  Detail & Related papers  (2021-02-07T20:53:23Z)
• Zeroth-Order Supervised Policy Improvement [94.0748002906652]
  Policy gradient (PG) algorithms have been widely used in reinforcement learning (RL) We propose Zeroth-Order Supervised Policy Improvement (ZOSPI) ZOSPI exploits the estimated value function $Q$ globally while preserving the local exploitation of the PG methods.
  arXiv  Detail & Related papers  (2020-06-11T16:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.