Agent Safety Should Be a Runtime Contract

• URL: http://arxiv.org/abs/2608.11274v1
• Date: Tue, 11 Aug 2026 08:01:05 GMT
• Title: Agent Safety Should Be a Runtime Contract
• Abstract summary: Agent safety should be a runtime contract enforced by the harness.<n> preventive face blocks dangerous actions before they happen.<n> evidential face requires verifiable proof that good actions actually happened.
• Score: 7.813225070853552
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The dominant paradigm treats AI safety as a property to be instilled during model training via RLHF, DPO, or Constitutional AI. We argue this is structurally insufficient for autonomous agents that execute code, mutate files, send messages, and modify databases. Agent safety should be a runtime contract enforced by the harness, and the contract has two complementary faces. The preventive face blocks dangerous actions before they happen via sandboxes, permission gates, output filters, and trajectory monitors. The evidential face requires verifiable proof that good actions actually happened, gating task submission on hard evidence such as test runs, log captures, file diffs, and citation grounding. We ground the position in four lines of public evidence, with row-level protocols and data released in the supplementary JSON files: a survey of 52 documented AI-agent and LLM safety incidents, a false-completion audit with 31 non-contested core cases plus one disputed illustrative case, a trajectory-schema audit of 12 public agent systems and harnesses, and a title-level audit of all 28,560 papers accepted at NeurIPS, ICML, and ICLR 2023-2025 showing a pooled 8-12x imbalance between training-time and deployment-time publication. Two prior communities that needed to enforce safety, computer security and the experimental sciences, converged on runtime contracts with both preventive and evidential elements; agentic AI is now under the same pressure. We formalize an Agent Trajectory Schema and Evidence Chain, state a compositional gating proposition based on standard monitor composition, and outline a research agenda. The right unit of safety in agentic AI is the trajectory-with-checkable-evidence, not the model.

Related papers

• The Agent Incident Registry: Toward Preventing Repeated AI Agent Failures [0.0]
  We present the Agent Incident Registry (AIR), a source-linked catalog containing 487 records of agent-related events disclosed from 2022 through 2026.<n>AIR supports source-grounded case retrieval and evaluation-scope auditing, not failure-rate or control-efficacy estimation.
  arXiv  Detail & Related papers  (2026-09-10T03:20:46Z)
• A Black Box for Agentic Processes: Blockchain-Anchored Evidence for AI Agent Communication, Human Oversight, and GRC Audits [0.0]
  Motivated by the 2026 OpenAI/Hugging Face incident, this position and architecture paper proposes a product- and vendor-neutral black-box architecture for agentic processes.<n>The architecture creates blockchain-anchored cryptographic commitments for selected agent communications, human-in-the-loop approvals, tool calls, and process artifacts without placing sensitive content on-chain.
  arXiv  Detail & Related papers  (2026-09-03T15:55:39Z)
• Zero-Knowledge Predicate Proofs Between AI Agents: A Measured, Cross-Protocol Gateway and the Source-Integrity Gap [0.0]
  Multi-agent AI platforms move quickly from staging to production, but the way agents establish trust remains rudimentary.<n>We take provable data minimisation between agents from proposal to running system.<n>A 32-bit threshold predicate proves in 6.2 ms and verifies in 1.0 ms with a 608-byte Bulletproofs proof.
  arXiv  Detail & Related papers  (2026-08-30T23:16:17Z)
• Distributing Security Controls Through Harness Engineering [0.0]
  Security and risk concerns remain the primary barrier to scaling agentic AI across organizations.<n>This paper investigates whether off-the-shelf security controls can be implemented on commercial AI coding agents.
  arXiv  Detail & Related papers  (2026-07-28T15:50:16Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• ToolAlignBench: Investigating Alignment Conflicts in Tool-Calling Enabled LLMs [11.599619860803472]
  We find that safety-aligned open-source models override their deployment instructions up to 43.4% of the time.<n>We also find that abliteration reduces rates of external whistleblowing.<n>These results reveal a fundamental tension in pluralistic alignment, where the same safety training that protects users can cause agents to act against deployment instructions in ways that create unpredictable liability risks.
  arXiv  Detail & Related papers  (2026-07-15T18:48:49Z)
• The Balkanization of Execution-Security Research for AI Coding Agents: Isolation, Access Control, and Time-of-Check-to-Time-of-Use Vulnerabilities [0.0]
  Papers on sandbox isolation, capability and access control, policy enforcement, time-of-check-to-time-of-use (TOCTOU) races, Model Context Protocol (MCP) threats, identity delegation, execution provenance, network egress control, and static analysis of agent-generated code are published independently and rarely cite one another.<n>We systematize 39 papers published between 2023 and 2026 into 17 categories, each verified directly against its source.<n>Reading across categories surfaces five cross-cutting gaps that no single paper addresses.
  arXiv  Detail & Related papers  (2026-07-07T01:56:18Z)
• VASO: Formally Verifiable Self-Evolving Skills for Physical AI Agents [57.240036084348354]
  We introduce VASO, a framework for verification-guided self-evolution of robot skill contracts.<n>VASO verifies logically inconsistent skill contracts, then verifies plans induced by the skill against global and local temporal specifications.<n>On Clearpath Jackal and PX4 quadcopter tasks, VASO reaches 97.2% formal-specification compliance using fewer than 100 optimization samples.
  arXiv  Detail & Related papers  (2026-06-03T20:02:35Z)
• When the Agent Is the Adversary: Architectural Requirements for Agentic AI Containment After the April 2026 Frontier Model Escape [0.0]
  This paper analyzes four categories of current containment approaches.<n>We categorize five behavioral incidents from the public disclosure and situate them within 698 real-world AI scheming incidents.
  arXiv  Detail & Related papers  (2026-04-25T19:41:00Z)
• Agents of Chaos [50.53354213047402]
  We report an exploratory red-teaming study of autonomous language-model-powered agents deployed in a live laboratory environment.<n>Twenty AI researchers interacted with the agents under benign and adversarial conditions.<n>Our findings establish the existence of security-, privacy-, and governance-relevant vulnerabilities in realistic deployment settings.
  arXiv  Detail & Related papers  (2026-02-23T16:28:48Z)
• OMNI-LEAK: Orchestrator Multi-Agent Network Induced Data Leakage [59.3826294523924]
  We investigate the security vulnerabilities of a popular multi-agent pattern known as the orchestrator setup.<n>We report the susceptibility of frontier models to different categories of attacks, finding that both reasoning and non-reasoning models are vulnerable.
  arXiv  Detail & Related papers  (2026-02-13T21:32:32Z)
• Adaptive Attacks on Trusted Monitors Subvert AI Control Protocols [80.68060125494645]
  We study adaptive attacks by an untrusted model that knows the protocol and the monitor model.<n>We instantiate a simple adaptive attack vector by which the attacker embeds publicly known or zero-shot prompt injections in the model outputs.
  arXiv  Detail & Related papers  (2025-10-10T15:12:44Z)
• Security Challenges in AI Agent Deployment: Insights from a Large Scale Public Competition [101.86739402748995]
  We run the largest public red-teaming competition to date, targeting 22 frontier AI agents across 44 realistic deployment scenarios.<n>We build the Agent Red Teaming benchmark and evaluate it across 19 state-of-the-art models.<n>Our findings highlight critical and persistent vulnerabilities in today's AI agents.
  arXiv  Detail & Related papers  (2025-07-28T05:13:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.