Distributional AGI Safety

• URL: http://arxiv.org/abs/2512.16856v1
• Date: Thu, 18 Dec 2025 18:29:50 GMT
• Title: Distributional AGI Safety
• Abstract summary: We argue that this patchwork AGI hypothesis needs to be given serious consideration.<n>We propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents.
• Score: 8.680471071138017
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI safety and alignment research has predominantly been focused on methods for safeguarding individual AI systems, resting on the assumption of an eventual emergence of a monolithic Artificial General Intelligence (AGI). The alternative AGI emergence hypothesis, where general capability levels are first manifested through coordination in groups of sub-AGI individual agents with complementary skills and affordances, has received far less attention. Here we argue that this patchwork AGI hypothesis needs to be given serious consideration, and should inform the development of corresponding safeguards and mitigations. The rapid deployment of advanced AI agents with tool-use capabilities and the ability to communicate and coordinate makes this an urgent safety consideration. We therefore propose a framework for distributional AGI safety that moves beyond evaluating and aligning individual agents. This framework centers on the design and implementation of virtual agentic sandbox economies (impermeable or semi-permeable), where agent-to-agent transactions are governed by robust market mechanisms, coupled with appropriate auditability, reputation management, and oversight to mitigate collective risks.

Related papers

• Towards trustworthy agentic AI: a comprehensive survey of safety, robustness, privacy, and system security [57.35851886874902]
  Agentic AI systems execute complex tasks autonomously, but their multi-step trajectories introduce new failure modes that challenge trustworthiness.<n>This survey provides a focused examination of trustworthy agentic AI through two core dimensions that are critical for high-risk deployments.<n>For each dimension, we clarify key concepts, identify where risks emerge along the agent workflow, and summarize stage-targeted mitigation strategies.
  arXiv  Detail & Related papers  (2026-05-17T10:26:37Z)
• AgentSOC: A Multi-Layer Agentic AI Framework for Security Operations Automation [0.0]
  This study introduces AgentSOC, a multi-layered agentic AI framework that enhances SOC automation by integrating perception, anticipatory reasoning, and risk-based action planning.<n>The proposed architecture consolidates several layers of abstraction to provide a single operational loop to support normalizing alerts, enriching context, generating hypotheses, validating structural feasibility, and executing policy-compliant responses.
  arXiv  Detail & Related papers  (2026-04-22T03:01:03Z)
• OMNI-LEAK: Orchestrator Multi-Agent Network Induced Data Leakage [59.3826294523924]
  We investigate the security vulnerabilities of a popular multi-agent pattern known as the orchestrator setup.<n>We report the susceptibility of frontier models to different categories of attacks, finding that both reasoning and non-reasoning models are vulnerable.
  arXiv  Detail & Related papers  (2026-02-13T21:32:32Z)
• AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security [126.49733412191416]
  Current guardrail models lack agentic risk awareness and transparency in risk diagnosis.<n>We propose a unified three-dimensional taxonomy that categorizes agentic risks by their source (where), failure mode (how), and consequence (what)<n>We introduce a new fine-grained agentic safety benchmark (ATBench) and a Diagnostic Guardrail framework for agent safety and security (AgentDoG)
  arXiv  Detail & Related papers  (2026-01-26T13:45:41Z)
• Adaptation of Agentic AI [162.63072848575695]
  We unify the rapidly expanding research landscape into a systematic framework that spans both agent adaptations and tool adaptations.<n>We demonstrate that this framework helps clarify the design space of adaptation strategies in agentic AI.<n>We then review the representative approaches in each category, analyze their strengths and limitations, and highlight key open challenges and future opportunities.
  arXiv  Detail & Related papers  (2025-12-18T08:38:51Z)
• AURA: An Agent Autonomy Risk Assessment Framework [0.0]
  AURA (Agent aUtonomy Risk Assessment) is a unified framework designed to detect, quantify, and mitigate risks arising from agentic AI.<n>AURA provides an interactive process to score, evaluate and mitigate the risks of running one or multiple AI Agents, synchronously or asynchronously.<n>AURA supports a responsible and transparent adoption of agentic AI and provides robust risk detection and mitigation while balancing computational resources.
  arXiv  Detail & Related papers  (2025-10-17T15:30:29Z)
• Governable AI: Provable Safety Under Extreme Threat Models [31.36879992618843]
  We propose a Governable AI (GAI) framework that shifts from traditional internal constraints to externally enforced structural compliance.<n>The GAI framework is composed of a simple yet reliable, fully deterministic, powerful, flexible, and general-purpose rule enforcement module (REM); governance rules; and a governable secure super-platform (GSSP) that offers end-to-end protection against compromise or subversion by AI.
  arXiv  Detail & Related papers  (2025-08-28T04:22:59Z)
• Never Compromise to Vulnerabilities: A Comprehensive Survey on AI Governance [211.5823259429128]
  We propose a comprehensive framework integrating technical and societal dimensions, structured around three interconnected pillars: Intrinsic Security, Derivative Security, and Social Ethics.<n>We identify three core challenges: (1) the generalization gap, where defenses fail against evolving threats; (2) inadequate evaluation protocols that overlook real-world risks; and (3) fragmented regulations leading to inconsistent oversight.<n>Our framework offers actionable guidance for researchers, engineers, and policymakers to develop AI systems that are not only robust and secure but also ethically aligned and publicly trustworthy.
  arXiv  Detail & Related papers  (2025-08-12T09:42:56Z)
• TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systems [8.683314804719506]
  This review presents a structured analysis of Trust, Risk, and Security Management (TRiSM) in the context of Agentic Multi-Agent Systems (AMAS)<n>We begin by examining the conceptual foundations of Agentic AI and highlight its architectural distinctions from traditional AI agents.<n>We then adapt and extend the AI TRiSM framework for Agentic AI, structured around key pillars: textit Explainability, ModelOps, Security, Privacy and textittheir lifecycle governance<n>A risk taxonomy is proposed to capture the unique threats and vulnerabilities of Agentic AI, ranging from coordination failures to
  arXiv  Detail & Related papers  (2025-06-04T16:26:11Z)
• LLM Agents Should Employ Security Principles [60.03651084139836]
  This paper argues that the well-established design principles in information security should be employed when deploying Large Language Model (LLM) agents at scale.<n>We introduce AgentSandbox, a conceptual framework embedding these security principles to provide safeguards throughout an agent's life-cycle.
  arXiv  Detail & Related papers  (2025-05-29T21:39:08Z)
• A Novel Zero-Trust Identity Framework for Agentic AI: Decentralized Authentication and Fine-Grained Access Control [7.228060525494563]
  This paper posits the imperative for a novel Agentic AI IAM framework.<n>We propose a comprehensive framework built upon rich, verifiable Agent Identities (IDs)<n>We also explore how Zero-Knowledge Proofs (ZKPs) enable privacy-preserving attribute disclosure and verifiable policy compliance.
  arXiv  Detail & Related papers  (2025-05-25T20:21:55Z)
• Safeguarding AI Agents: Developing and Analyzing Safety Architectures [0.0]
  This paper addresses the need for safety measures in AI systems that collaborate with human teams.<n>We propose and evaluate three frameworks to enhance safety protocols in AI agent systems.<n>We conclude that these frameworks can significantly strengthen the safety and security of AI agent systems.
  arXiv  Detail & Related papers  (2024-09-03T10:14:51Z)
• EARBench: Towards Evaluating Physical Risk Awareness for Task Planning of Foundation Model-based Embodied AI Agents [53.717918131568936]
  Embodied artificial intelligence (EAI) integrates advanced AI models into physical entities for real-world interaction.<n>Foundation models as the "brain" of EAI agents for high-level task planning have shown promising results.<n>However, the deployment of these agents in physical environments presents significant safety challenges.<n>This study introduces EARBench, a novel framework for automated physical risk assessment in EAI scenarios.
  arXiv  Detail & Related papers  (2024-08-08T13:19:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.