GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization

• URL: http://arxiv.org/abs/2604.00717v1
• Date: Wed, 01 Apr 2026 10:26:22 GMT
• Title: GRASP: Gradient Realignment via Active Shared Perception for Multi-Agent Collaborative Optimization
• Abstract summary: Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations.<n>We propose Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution.
• Score: 41.52202306408042
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Non-stationarity arises from concurrent policy updates and leads to persistent environmental fluctuations. Existing approaches like Centralized Training with Decentralized Execution (CTDE) and sequential update schemes mitigate this issue. However, since the perception of the policies of other agents remains dependent on sampling environmental interaction data, the agent essentially operates in a passive perception state. This inevitably triggers equilibrium oscillations and significantly slows the convergence speed of the system. To address this issue, we propose Gradient Realignment via Active Shared Perception (GRASP), a novel framework that defines generalized Bellman equilibrium as a stable objective for policy evolution. The core mechanism of GRASP involves utilizing the independent gradients of agents to derive a defined consensus gradient, enabling agents to actively perceive policy updates and optimize team collaboration. Theoretically, we leverage the Kakutani Fixed-Point Theorem to prove that the consensus direction $u^*$ guarantees the existence and attainability of this equilibrium. Extensive experiments on StarCraft II Multi-Agent Challenge (SMAC) and Google Research Football (GRF) demonstrate the scalability and promising performance of the framework.

Related papers

• ACPO: Agent-Chained Policy Optimization for Multi-Agent Reinforcement Learning [47.96591344642519]
  Multi-Agent Reinforcement Learning requires agents to collectively maximize a shared return.<n>Prior methods largely follow two approaches: independent factorized updates with centralized critics, or alternating best-response updates.<n>We show the joint policy gradient admits an exact decentralized decomposition of per-agent terms, each formed from per-agent score functions and decentralized critics.<n>Central to this result is a serialized view of the simultaneous joint decision in which agents commit actions one at a time, each conditioning on a belief over preceding actions.
  arXiv  Detail & Related papers  (2026-06-29T10:04:15Z)
• Self-evolving LLM agents with in-distribution Optimization [60.05867547965365]
  Large Language Models (LLMs) have recently emerged as powerful controllers for interactive agents in complex environments.<n>We propose Q-Evolve, a self-evolving framework for LLM agents that unifies automatic process-reward labeling and policy learning.<n>We evaluate our method on AlfWorld, WebShop, and ScienceWorld, showing Q-Evolve outperforms strong baselines in sample efficiency, robustness, and overall task performance.
  arXiv  Detail & Related papers  (2026-06-05T15:09:52Z)
• Equilibrium Selection in Multi-Agent Policy Gradients via Opponent-Aware Basin Entry [0.0]
  We study basin-entry probability with respect to a target set of equilibria selected by an external criterion.<n>For finite-unroll Meta-MAPG, we show that the update decomposes into ordinary policy gradient plus own-learning and peer-learning corrections.
  arXiv  Detail & Related papers  (2026-05-18T08:57:15Z)
• HALyPO: Heterogeneous-Agent Lyapunov Policy Optimization for Human-Robot Collaboration [37.15750888195892]
  We propose heterogeneous-agent Lyapunov policy optimization (HALyPO)<n>HALyPO uses Lyapunov certification to stabilize decentralized policy learning.<n>We show that this certified stability improves generalization and robustness in collaborative corner cases.
  arXiv  Detail & Related papers  (2026-03-04T05:26:13Z)
• Socially-Weighted Alignment: A Game-Theoretic Framework for Multi-Agent LLM Systems [17.658093330392052]
  We propose a game-theoretic framework that modifies inference-time decision making by interpolating between an agent's private objective and an estimate of group welfare.<n>We show that SWA induces a critical threshold $*=(n-)/(n-1)$ above which agents no longer have marginal incentive to increase demand under overload.
  arXiv  Detail & Related papers  (2026-02-16T05:17:58Z)
• GRPO-Guard: Mitigating Implicit Over-Optimization in Flow Matching via Regulated Clipping [63.33669214116784]
  GRPO-Guard is a simple yet effective enhancement to existing GRPO frameworks.<n>It restores a balanced and step-consistent importance ratio, ensuring that PPO clipping properly constrains harmful updates.<n>It substantially mitigates implicit over-optimization without relying on heavy KL regularization.
  arXiv  Detail & Related papers  (2025-10-25T14:51:17Z)
• Iterative Refinement of Flow Policies in Probability Space for Online Reinforcement Learning [56.47948583452555]
  We introduce the Stepwise Flow Policy (SWFP) framework, founded on the key insight that discretizing the flow matching inference process via a fixed-step Euler scheme aligns it with the variational Jordan-Kinderlehrer-Otto principle from optimal transport.<n>SWFP decomposes the global flow into a sequence of small, incremental transformations between proximate distributions.<n>This decomposition yields an efficient algorithm that fine-tunes pre-trained flows via a cascade of small flow blocks, offering significant advantages.
  arXiv  Detail & Related papers  (2025-10-17T07:43:51Z)
• Towards Fast Rates for Federated and Multi-Task Reinforcement Learning [34.34798425737858]
  We propose Fast-FedPG, a novel federated policy algorithm with a carefully designed bias-correction mechanism. Under a gradient-domination condition, we prove that our algorithm guarantees (i) fast linear convergence with exact gradients, and (ii) sub-linear rates that enjoy a linear speedup w.r.t. the number of agents with noisy, truncated policy gradients.
  arXiv  Detail & Related papers  (2024-09-09T02:59:17Z)
• Inducing Stackelberg Equilibrium through Spatio-Temporal Sequential Decision-Making in Multi-Agent Reinforcement Learning [17.101534531286298]
  We construct a Nash-level policy model based on a conditional hypernetwork shared by all agents. This approach allows for asymmetric training with symmetric execution, with each agent responding optimally conditioned on the decisions made by superior agents. Experiments demonstrate that our method effectively converges to the SE policies in repeated matrix game scenarios.
  arXiv  Detail & Related papers  (2023-04-20T14:47:54Z)
• Faster Last-iterate Convergence of Policy Optimization in Zero-Sum Markov Games [63.60117916422867]
  This paper focuses on the most basic setting of competitive multi-agent RL, namely two-player zero-sum Markov games. We propose a single-loop policy optimization method with symmetric updates from both agents, where the policy is updated via the entropy-regularized optimistic multiplicative weights update (OMWU) method. Our convergence results improve upon the best known complexities, and lead to a better understanding of policy optimization in competitive Markov games.
  arXiv  Detail & Related papers  (2022-10-03T16:05:43Z)
• Implicit Distributional Reinforcement Learning [61.166030238490634]
  implicit distributional actor-critic (IDAC) built on two deep generator networks (DGNs) Semi-implicit actor (SIA) powered by a flexible policy distribution. We observe IDAC outperforms state-of-the-art algorithms on representative OpenAI Gym environments.
  arXiv  Detail & Related papers  (2020-07-13T02:52:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.