Soft-Label Governance for Distributional Safety in Multi-Agent Systems

• URL: http://arxiv.org/abs/2604.19752v1
• Date: Thu, 19 Mar 2026 00:56:09 GMT
• Title: Soft-Label Governance for Distributional Safety in Multi-Agent Systems
• Abstract summary: Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation.<n>We introduce SWARM, a simulation framework that replaces binary good/bad labels with emphsoft probabilistic labels.<n>In seven scenarios with five-seed replication, strict governance reduces welfare by over 40% without improving safety.
• Score: 4.726777092009554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent AI systems exhibit emergent risks that no single agent produces in isolation. Existing safety frameworks rely on binary classifications of agent behavior, discarding the uncertainty inherent in proxy-based evaluation. We introduce SWARM (\textbf{S}ystem-\textbf{W}ide \textbf{A}ssessment of \textbf{R}isk in \textbf{M}ulti-agent systems), a simulation framework that replaces binary good/bad labels with \emph{soft probabilistic labels} $p = P(v{=}+1) \in [0,1]$, enabling continuous-valued payoff computation, toxicity measurement, and governance intervention. SWARM implements a modular governance engine with configurable levers (transaction taxes, circuit breakers, reputation decay, and random audits) and quantifies their effects through probabilistic metrics including expected toxicity $\mathbb{E}[1{-}p \mid \text{accepted}]$ and quality gap $\mathbb{E}[p \mid \text{accepted}] - \mathbb{E}[p \mid \text{rejected}]$. Across seven scenarios with five-seed replication, strict governance reduces welfare by over 40\% without improving safety. In parallel, aggressively internalizing system externalities collapses total welfare from a baseline of $+262$ down to $-67$, while toxicity remains invariant. Circuit breakers require careful calibration; overly restrictive thresholds severely diminish system value, whereas an optimal threshold balances moderate welfare with minimized toxicity. Companion experiments show soft metrics detect proxy gaming by self-optimizing agents passing conventional binary evaluations. This basic governance layer applies to live LLM-backed agents (Concordia entities, Claude, GPT-4o Mini) without modification. Results show distributional safety requires \emph{continuous} risk metrics and governance lever calibration involves quantifiable safety-welfare tradeoffs. Source code and project resources are publicly available at https://www.swarm-ai.org/.

Related papers

• BAPR: Bayesian amnesic piecewise-robust reinforcement learning for non-stationary continuous control [19.34848029403215]
  Real-world control systems operate under emphpiecewise stationary conditions, where dynamics remain stable for extended periods before undergoing regime changes.<n>Standard robust RL methods face a fundamental dilemma: a globally conservative policy wastes performance during stable periods, while a locally adaptive policy risks catastrophic failure when the regime changes undetected.<n>We propose textbfBAPR, which unifies Bayesian Online Change Detection with robust ensemble RL.
  arXiv  Detail & Related papers  (2026-05-15T16:49:05Z)
• Governing What You Cannot Observe: Adaptive Runtime Governance for Autonomous AI Agents [0.0]
  Autonomous AI agents can remain fully authorized and still become unsafe as behavior drifts, adversaries adapt, and decision patterns shift without any code change.<n>We propose the textbfInformational Viability Principle: governing an agent reduces to estimating a bound on unobserved risk.<n>textbfRiskGate instantiates the framework with dedicated statistical estimators (KL divergence, segment-vs-rest $z$-tests, sequential pattern matching), a fail-secure monotonic pipeline, and a closed-loop Autopilot.
  arXiv  Detail & Related papers  (2026-04-27T16:46:15Z)
• Optimistic Policy Learning under Pessimistic Adversaries with Regret and Violation Guarantees [6.310266319378214]
  Real-world decision-making systems operate in environments where state transitions depend on the agent's actions.<n>Standard Constrained MDP formulations assume the agent is the sole driver of state evolution.<n>We propose textbfRobust Hallucinated Constrained Upper-Confidence RL (textttRHC-UCRL), a model-based algorithm that maintains optimism over both agent and adversary policies.
  arXiv  Detail & Related papers  (2026-04-15T04:53:29Z)
• Tight Convergence Rates for Online Distributed Linear Estimation with Adversarial Measurements [66.94250413799232]
  We study mean estimation of a random vector $X$ in a distributed parameter-server-worker setup.<n>Key challenges are adversarial measurements and asynchrony.<n>Results provide a unified finite-time characterization of robustness, identifiability, and statistical efficiency in distributed linear estimation with adversarial workers.
  arXiv  Detail & Related papers  (2026-04-07T11:45:55Z)
• Learning operators on labelled conditional distributions with applications to mean field control of non exchangeable systems [3.441021278275805]
  We study the approximation of operators acting on probability measures on a product space with prescribed marginal.<n>Our main theoretical result establishes a universal approximation theorem for continuous operators on $cal M_$.<n>We introduce a sampling strategy for generating training measures in $cal M_$, enabling practical learning of such conditional mean-field operators.
  arXiv  Detail & Related papers  (2026-03-23T08:13:21Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• Socially-Weighted Alignment: A Game-Theoretic Framework for Multi-Agent LLM Systems [17.658093330392052]
  We propose a game-theoretic framework that modifies inference-time decision making by interpolating between an agent's private objective and an estimate of group welfare.<n>We show that SWA induces a critical threshold $*=(n-)/(n-1)$ above which agents no longer have marginal incentive to increase demand under overload.
  arXiv  Detail & Related papers  (2026-02-16T05:17:58Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Phase Transition for Budgeted Multi-Agent Synergy [41.486076708302456]
  Multi-agent systems can improve reliability, yet under a fixed inference budget they often help, saturate, or even collapse.<n>We develop a minimal and calibratable theory that predicts these regimes from three binding constraints of modern agent stacks.
  arXiv  Detail & Related papers  (2026-01-24T05:32:50Z)
• UCB-type Algorithm for Budget-Constrained Expert Learning [71.67657715154034]
  algnameM-LCB is a UCB-style meta-algorithm that provides emphanytime regret guarantees<n>We show how algnameM-LCB extends the classical bandit paradigm to the more realistic scenario of coordinating stateful, self-learning experts under limited resources.
  arXiv  Detail & Related papers  (2025-10-26T12:36:17Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Contextual Combinatorial Bandits with Probabilistically Triggered Arms [55.9237004478033]
  We study contextual bandits with probabilistically triggered arms (C$2$MAB-T) under a variety of smoothness conditions. Under the triggering modulated (TPM) condition, we devise the C$2$-UC-T algorithm and derive a regret bound $tildeO(dsqrtT)$.
  arXiv  Detail & Related papers  (2023-03-30T02:51:00Z)
• Provably Efficient Safe Exploration via Primal-Dual Policy Optimization [105.7510838453122]
  We study the Safe Reinforcement Learning (SRL) problem using the Constrained Markov Decision Process (CMDP) formulation. We present an provably efficient online policy optimization algorithm for CMDP with safe exploration in the function approximation setting.
  arXiv  Detail & Related papers  (2020-03-01T17:47:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.