Co2PO: Coordinated Constrained Policy Optimization for Multi-Agent RL

• URL: http://arxiv.org/abs/2602.02970v1
• Date: Tue, 03 Feb 2026 01:09:31 GMT
• Title: Co2PO: Coordinated Constrained Policy Optimization for Multi-Agent RL
• Abstract summary: We propose Co2PO, a novel framework that enables coordination-driven safety through selective, risk-aware communication.<n>Co2PO introduces a shared blackboard architecture for broadcasting positional intent and yield signals, governed by a learned hazard predictor.<n>We evaluate Co2PO across a suite of complex multi-agent safety benchmarks, where it achieves higher returns compared to leading constrained baselines.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Constrained multi-agent reinforcement learning (MARL) faces a fundamental tension between exploration and safety-constrained optimization. Existing leading approaches, such as Lagrangian methods, typically rely on global penalties or centralized critics that react to violations after they occur, often suppressing exploration and leading to over-conservatism. We propose Co2PO, a novel MARL communication-augmented framework that enables coordination-driven safety through selective, risk-aware communication. Co2PO introduces a shared blackboard architecture for broadcasting positional intent and yield signals, governed by a learned hazard predictor that proactively forecasts potential violations over an extended temporal horizon. By integrating these forecasts into a constrained optimization objective, Co2PO allows agents to anticipate and navigate collective hazards without the performance trade-offs inherent in traditional reactive constraints. We evaluate Co2PO across a suite of complex multi-agent safety benchmarks, where it achieves higher returns compared to leading constrained baselines while converging to cost-compliant policies at deployment. Ablation studies further validate the necessity of risk-triggered communication, adaptive gating, and shared memory components.

Related papers

• Think2Go: Generative Next POI Recommendation with LLM Reasoning [57.37706163990998]
  Next Point-of-Interest (POI) recommendation task focuses on mining user behavioral preference patterns from historical check-ins to provide personalized suggestions for the next destination.<n>Existing methods rely on shallow contextual information and handcrafted feature interactions to predict the next POI.<n>We propose Think2Go, a novel generative next POI recommendation framework, which enhances the model's comprehension of SID representations and explores diverse spatial-temporal patterns via test-time computational scaling.
  arXiv  Detail & Related papers  (2026-07-31T03:52:20Z)
• CSPO: Constraint-Sensitive Policy Optimization for Safe Reinforcement Learning [9.011823422505378]
  We propose Constraint-Sensitive Policy Optimization (CSPO), a first-order primal-dual method that incorporates local constraint sensitivity into policy updates.<n>CSPO augments the primal objective with a constraint-sensitive correction derived from the shortest signed distance to the safety boundary, enabling smarter recovery steps back to safety.<n> Experiments on navigation and locomotion benchmarks demonstrate that CSPO achieves faster safety recovery and high reward preservation, resulting in higher constrained returns.
  arXiv  Detail & Related papers  (2026-06-12T12:48:56Z)
• Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observability [79.08785366532287]
  We propose a lightweight risk-gated reinforcement learning approximation for risk-sensitive control under partial observability.<n>We evaluate the approach in two safety-critical partially observable domains: automated glucose regulation and safety-constrained navigation.
  arXiv  Detail & Related papers  (2026-05-14T01:23:09Z)
• Rethinking Ratio-Based Trust Regions for Policy Optimization in Multi-Agent Reinforcement Learning [8.199581456016062]
  Ratio-based trust-region methods update decentralized actors using per-agent probability ratios weighted by joint advantage estimates.<n>We introduce Multi-Agent Ratio Symmetry (MARS), a novel policy optimization objective that replaces these additive ratio-based trust-region mechanisms with a multiplicatively symmetric geometric barrier.<n>MARS matches or exceeds MAPPO and MASPO in aggregate environment-level performance.
  arXiv  Detail & Related papers  (2026-05-09T23:14:27Z)
• Safe and Policy-Compliant Multi-Agent Orchestration for Enterprise AI [0.0]
  This paper introduces 92-97% C (Constraint-Aware Multi-Agent Cognitive Orchestration)<n>C integrates three mechanisms: (i) a constraint projection engine enforcing policy-feasible actions via convex projection, (ii) adaptive risk-weighted Lagrangian utility shaping, and (iii) an iterative negotiation protocol with provably bounded convergence.<n>C operates as constrained deployment-time compatible with any agent architecture, with policy predicates designed for direct integration with production engines such as OPA.
  arXiv  Detail & Related papers  (2026-04-19T04:02:17Z)
• DARC: Disagreement-Aware Alignment via Risk-Constrained Decoding [59.16244104797919]
  We propose **Disagreement-Aware Alignment via Risk-Constrained Decoding (DARC)**.<n>DARC frames response selection as distributionally robust, risk-sensitive decision making.<n>Experiments on alignment benchmarks show that DARC reduces disagreement and tail risk while maintaining competitive average quality.
  arXiv  Detail & Related papers  (2026-03-09T09:21:29Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search [72.87861928940929]
  Boundary-Aware Policy Optimization (BAPO) is a novel RL framework designed to cultivate reliable boundary awareness without compromising accuracy.<n>BAPO introduces two key components: (i) a group-based boundary-aware reward that encourages an IDK response only when the reasoning reaches its limit, and (ii) an adaptive reward modulator that strategically suspends this reward during early exploration, preventing the model from exploiting IDK as a shortcut.
  arXiv  Detail & Related papers  (2026-01-16T07:06:58Z)
• MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization [56.074760766965085]
  Group-Relative Policy Optimization has emerged as an efficient paradigm for aligning Large Language Models (LLMs)<n>We propose MAESTRO, which treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck.<n>We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal.
  arXiv  Detail & Related papers  (2026-01-12T05:02:48Z)
• Safe, Efficient, and Robust Reinforcement Learning for Ranking and Diffusion Models [2.231476498067998]
  dissertation investigates how reinforcement learning methods can be designed to be safe, sample-efficient, and robust.<n> Framed through the unifying perspective of contextual-bandit RL, the work addresses two major application domains - ranking and recommendation, and text-to-image diffusion models.
  arXiv  Detail & Related papers  (2025-10-17T08:37:38Z)
• Optimism as Risk-Seeking in Multi-Agent Reinforcement Learning [3.2693858240177565]
  We propose a principled framework that interprets risk-seeking objectives as optimism.<n>We derive a policy-gradient theorem for optimistic value functions, including explicit formulas for the entropic risk/KL-penalty setting.<n>Our framework thus unifies risk-sensitive learning and optimism, offering a theoretically grounded and practically effective approach to cooperation in MARL.
  arXiv  Detail & Related papers  (2025-09-28T19:44:59Z)
• Multi-Agent Trust Region Policy Optimisation: A Joint Constraint Approach [17.48210470289556]
  Heterogeneous-Agent Trust Region Policy Optimization (HATRPO) enforces per-agent trust region constraints using Kullback-Leibler (KL) divergence to stabilize training.<n> assigning each agent the same KL threshold can lead to slow and locally optimal updates, especially in heterogeneous settings.<n>We propose two approaches for allocating the KL divergence threshold across agents: HATRPO-W, a Karush-Kuhn-Tucker-based (KKT-based) method that optimize threshold assignment under global KL constraints, and HATRPO-G, a greedy algorithm that prioritizes agents based on improvement-to
  arXiv  Detail & Related papers  (2025-08-14T04:48:46Z)
• Embedding Safety into RL: A New Take on Trust Region Methods [1.5733417396701983]
  We introduce Constrained Trust Region Policy Optimization (C-TRPO), which reshapes policy space to ensure trust regions contain only safe policies.<n>Experiments show that C-TRPO reduces constraint violations while maintaining competitive returns.
  arXiv  Detail & Related papers  (2024-11-05T09:55:50Z)
• Last-Iterate Global Convergence of Policy Gradients for Constrained Reinforcement Learning [62.81324245896717]
  We introduce an exploration-agnostic algorithm, called C-PG, which exhibits global last-ite convergence guarantees under (weak) gradient domination assumptions. We numerically validate our algorithms on constrained control problems, and compare them with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-07-15T14:54:57Z)
• A Reductions Approach to Risk-Sensitive Reinforcement Learning with Optimized Certainty Equivalents [44.09686403685058]
  We study risk-sensitive RL where the goal is learn a history-dependent policy that optimize some risk measure of cumulative rewards.<n>We propose two meta-algorithms: one grounded in optimism and another based on policy gradients.<n>We empirically show that our algorithms learn the optimal history-dependent policy in a proof-of-concept MDP.
  arXiv  Detail & Related papers  (2024-03-10T21:45:12Z)
• Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applications [73.58451824894568]
  The widely adopted CMDP model constrains the risks in expectation, which makes room for dangerous behaviors in long-tail states. In safety-critical domains, such behaviors could lead to disastrous outcomes. We propose Objective Suppression, a novel method that adaptively suppresses the task reward maximizing objectives according to a safety critic.
  arXiv  Detail & Related papers  (2024-02-23T23:22:06Z)
• Guided Cooperation in Hierarchical Reinforcement Learning via Model-based Rollout [16.454305212398328]
  We propose a goal-conditioned hierarchical reinforcement learning (HRL) framework named Guided Cooperation via Model-based Rollout (GCMR) GCMR aims to bridge inter-layer information synchronization and cooperation by exploiting forward dynamics. Experimental results demonstrate that incorporating the proposed GCMR framework with a disentangled variant of HIGL, namely ACLG, yields more stable and robust policy improvement.
  arXiv  Detail & Related papers  (2023-09-24T00:13:16Z)
• Risk-Sensitive Deep RL: Variance-Constrained Actor-Critic Provably Finds Globally Optimal Policy [95.98698822755227]
  We make the first attempt to study risk-sensitive deep reinforcement learning under the average reward setting with the variance risk criteria. We propose an actor-critic algorithm that iteratively and efficiently updates the policy, the Lagrange multiplier, and the Fenchel dual variable.
  arXiv  Detail & Related papers  (2020-12-28T05:02:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.