SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do

• URL: http://arxiv.org/abs/2603.27148v1
• Date: Sat, 28 Mar 2026 05:52:04 GMT
• Title: SafetyDrift: Predicting When AI Agents Cross the Line Before They Actually Do
• Abstract summary: SafetyDrift models agent safety trajectories as absorbing Markov chains.<n>In communication tasks, agents that reach even a mild risk state have an 85% chance of violating safety within five steps.<n>A monitor built on these models detects 94.7% of violations with 3.7 steps of advance warning at negligible computational cost.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When an LLM agent reads a confidential file, then writes a summary, then emails it externally, no single step is unsafe, but the sequence is a data leak. We call this safety drift: individually safe actions compounding into violations. Prior work has measured this problem; we predict it. SafetyDrift models agent safety trajectories as absorbing Markov chains, computing the probability that a trajectory will reach a violation within a given number of steps via closed form absorption analysis. A consequence of the monotonic state design is that every agent will eventually violate safety if left unsupervised (absorption probability 1.0 from all states), making the practical question not if but when, and motivating our focus on finite horizon prediction. Across 357 traces spanning 40 realistic tasks in four categories, we discover that "points of no return" are sharply task dependent: in communication tasks, agents that reach even a mild risk state have an 85% chance of violating safety within five steps, while in technical tasks the probability stays below 5% from any state. A lightweight monitor built on these models detects 94.7% of violations with 3.7 steps of advance warning at negligible computational cost, outperforming both keyword matching (44.7% detection, 55.9% false positive rate) and per step LLM judges (52.6% detection, 38.2% false positive rate) while running over 60,000x faster.

Related papers

• StepGuard: Learning Step-Level Guardrails with Scalable Supervision and Safety-Utility Balancing [80.39644823192113]
  We propose StepGuard, a step-level guard model that can audit completed agent trajectories and check tool actions before they are executed.<n>StepGuard achieves the highest average accuracy among open-weight guard models, with performance comparable to GPT-5.4.<n>When used to guard agents on AgentDojo and AgentDyn, StepGuard reduces mean attack success rate by 77.3%.
  arXiv  Detail & Related papers  (2026-08-25T16:17:27Z)
• ClawSentry: A Progressive Multi-Tier Security Monitor for Safeguarding Autonomous LLM Agents [24.540638169054812]
  A single agent hijacked by a malicious third-party skill can cause data exfiltration, privilege escalation, or cascading compromise.<n>We present ClawSentry, an open-source, framework-agnostic security supervision gateway for agent runtimes.
  arXiv  Detail & Related papers  (2026-08-21T13:47:51Z)
• ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents [23.994328225666663]
  Cowork agents may complete benign tasks while disclosing protected data, manipulating unauthorized state, invocate unauthorized API.<n>We define behavioral safety and introduce ActBench, a benchmark that evaluates such behavior risk from execution trajectories rather than final responses.<n>ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs.
  arXiv  Detail & Related papers  (2026-08-10T11:45:03Z)
• AgentS4D: Benchmarking Runtime Risks across the Execution Lifecycle of LLM-Based Workspace Agents [10.99961131065026]
  Large language model (LLM)-based workspace agents execute stateful, multi-step across heterogeneous resources.<n>Recent benchmarks have advanced executable safety testing and trajectory-aware verification.<n>We introduce AgentS4D, a sandboxed benchmark for lifecycle-wide runtime safety evaluation.
  arXiv  Detail & Related papers  (2026-07-29T15:55:45Z)
• PaSBench-Video: A Streaming Video Benchmark for Proactive Safety Warning [54.70033525672316]
  Video-capable large language models (MLLMs) could serve as always-on safety monitors that issue warnings during this window.<n>We present PaSBench-Video, a 740-video benchmark with 481 risk and 259 no-risk videos across four domains.<n>We find that no model exceeds 20.0% on our strictest metric, and recall is tightly coupled with false-positive rate.
  arXiv  Detail & Related papers  (2026-06-01T16:14:01Z)
• On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment [54.30690671490447]
  Existing safety-alignment signals are largely response-level or off-policy.<n>FATE transforms verifier-scored failures into repair supervision without expert demonstrations.<n>FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%.
  arXiv  Detail & Related papers  (2026-05-12T09:56:28Z)
• Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Away [97.11976870616273]
  We propose a lightweight inference-time defense that treats safety recovery as a satising constraint rather than an objective.<n>In our evaluations across six open-source MLRMs and four jailbreak benchmarks, SafeThink reduces attack success rates by 30-60%.
  arXiv  Detail & Related papers  (2026-02-11T18:09:17Z)
• StepShield: When, Not Whether to Intervene on Rogue Agents [1.472404880217315]
  Existing agent safety benchmarks report binary accuracy, conflating early intervention with post-mortem analysis.<n>We introduce StepShield, the first benchmark to evaluate when violations are detected, not just whether.<n>By shifting the focus of evaluation from whether to when, StepShield provides a new foundation for building safer and more economically viable AI agents.
  arXiv  Detail & Related papers  (2026-01-29T18:55:46Z)
• Pro2Guard: Proactive Runtime Enforcement of LLM Agent Safety via Probabilistic Model Checking [8.970702398918924]
  Large Language Model (LLM) agents exhibit powerful autonomous capabilities across domains such as robotics, virtual assistants, and web automation.<n>Existing rule-based enforcement systems, such as AgentSpec, focus on developing reactive safety rules.<n>We propose Pro2Guard, a proactive runtime enforcement framework grounded in probabilistic reachability analysis.
  arXiv  Detail & Related papers  (2025-08-01T10:24:47Z)
• Towards Evaluating Proactive Risk Awareness of Multimodal Language Models [39.44421152112219]
  A proactive safety artificial intelligence (AI) system would work better than a reactive one.<n>PaSBench evaluates this capability through 416 multimodal scenarios.<n>Top performers like Gemini-2.5-pro achieve 71% image and 64% text accuracy, but miss 45-55% risks in repeated trials.
  arXiv  Detail & Related papers  (2025-05-23T04:28:47Z)
• When Uncertainty Leads to Unsafety: Empirical Insights into the Role of Uncertainty in Unmanned Aerial Vehicle Safety [9.432686404777584]
  This work investigates the relationship between the behavioral uncertainty of a UAV and the unsafety of its flight.<n>By quantifying uncertainty, it is possible to develop a predictor for unsafety, which acts as a flight supervisor.<n>Superialist achieved high performance in detecting uncertain behaviors with up to 96% precision and 93% recall.
  arXiv  Detail & Related papers  (2025-01-15T16:18:13Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.