HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses

• URL: http://arxiv.org/abs/2608.06984v1
• Date: Fri, 07 Aug 2026 09:03:49 GMT
• Title: HarnessSafe: Evaluating Safety Across Persistent Carriers in Agent Harnesses
• Abstract summary: harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts.<n>This capability creates delayed safety risks: attacker-influenced content can cross system boundaries and later affect the execution of a benign request.<n>We present HarnessSafe, a benchmark comprising 328 executable cases across seven persistent-carrier families and evaluated on most mainstream agent harnesses.
• Score: 20.04545361404688
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Modern agent harnesses persist state across tasks and sessions through persistent carriers like memory, skills, tools, and shared artifacts. However, this capability creates delayed safety risks: attacker-influenced content can cross system boundaries and later affect the execution of a benign request. Existing benchmarks typically focus on a few carriers or harnesses, while end-to-end attack-success rates reveal little about how risks propagate. To this end, we present HarnessSafe, a benchmark comprising 328 executable cases across seven persistent-carrier families and evaluated on most mainstream agent harnesses. Each case is specified as a Persistent-Risk Lifecycle that traces attacker influence from its initial entry, through persistence across carriers and system boundaries, to a later benign trigger and an observable violation. We further introduce a multi-stage, trace-based evaluation that uses observable execution evidence to determine how far each attack chain progresses and where it is stopped. Experiments show that containment is carrier-specific and strongly depends on the harness-model configuration. Both the harness and model backend substantially shape containment outcomes, while attack success rates cannot reflect distinct lifecycle progression patterns.

Related papers

• BLINDSPOT: A Benchmark for Safety and Refusal Calibration in Long-Horizon Tool-Using Agents [20.217145643468598]
  We introduce Blindspot, a benchmark for trajectory-level safety calibration of long-horizon tool-using agents.<n>Blindspot evaluates complete user-agent-environment trajectories through adaptive adversarial interaction, stateful tool execution, and execution-grounded adjudication.<n>We evaluate 13 proprietary and open-weight LLMs using eight metrics covering unsafe completion, appropriate refusal, benign utility, over-refusal, repeated-run robustness, and post-refusal failure.
  arXiv  Detail & Related papers  (2026-09-14T20:12:27Z)
• Reassembling Distributed Risk: Trajectory-Conditioned Action Generation for Multi-Turn Agent Safety [12.022506016268112]
  Under multi-turn decomposition attacks, a harmful objective can be distributed across individually plausible requests and tool calls.<n>We propose emphReassembling Distributed Risk (ReDiR), a generation-time defense that conditions action generation on trajectory-level security evidence.<n>We evaluate ReDiR on two agent-safety benchmarks across three model families and eight held-out tool domains.
  arXiv  Detail & Related papers  (2026-08-26T12:28:35Z)
• HarnessRisk: A Lifecycle-Oriented Benchmark for Agent Harness Safety [45.163264704448125]
  We present a lifecycle oriented benchmark that organizes agent safety into six operational phases.<n>We evaluate each trajectory using Utility, Attack Success Rate, Persistence, and Detection.<n>We find that explicit risk recognition does not reliably lead to safe action, as some configurations detect risks in more than 90% of runs.
  arXiv  Detail & Related papers  (2026-08-18T10:03:58Z)
• Self-Evolving Just-In-Time Memory for Proactive Embodied Safety [13.342067741014455]
  Existing safety approaches rely on runtime guardrails to block unsafe actions or induce excessive caution.<n>We introduce the Self-Evolving Just-In-Time Memory framework, which reframes embodied safety from progress-stalling guardrails to proactive hazard mitigation.
  arXiv  Detail & Related papers  (2026-06-26T14:26:24Z)
• Auditing Agent Harness Safety [81.22315979618612]
  LLM agents increasingly run inside execution harnesses that dispatch tools, allocate resources, and route messages between specialized components.<n>Most safety benchmarks score only final outputs or terminal states, even though many violations occur mid-trajectory rather than at termination.<n>We propose HarnessAudit, a framework that audits full execution trajectories across boundary compliance, execution fidelity, and system stability.
  arXiv  Detail & Related papers  (2026-05-14T02:14:28Z)
• SafeHarness: Lifecycle-Integrated Security Architecture for LLM-based Agent Deployment [19.947119280467934]
  safeharness is a security architecture in which four proposed defense layers are woven directly into the agent lifecycle.<n>We evaluate safeharness on benchmark datasets across diverse harness configurations.
  arXiv  Detail & Related papers  (2026-04-15T08:59:00Z)
• TraceSafe: A Systematic Assessment of LLM Guardrails on Multi-Step Tool-Calling Trajectories [20.868825285848196]
  Safety guardrails are well-benchmarked for natural language responses, but their efficacy remains largely unexplored within multi-step tool-use trajectories.<n>To address this gap, we introduce StructuralSafe-Bench, the first comprehensive benchmark specifically designed to assess mid-trajectory safety.
  arXiv  Detail & Related papers  (2026-04-08T15:46:14Z)
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails [74.63933201261595]
  Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks.<n>LRMs remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks.<n>We propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps.
  arXiv  Detail & Related papers  (2025-10-24T09:32:25Z)
• Building a Foundational Guardrail for General Agentic Systems via Synthetic Data [76.18834864749606]
  LLM agents can plan multi-step tasks, intervening at the planning stage-before any action is executed-is often the safest way to prevent harm.<n>Existing guardrails mostly operate post-execution, which is difficult to scale and leaves little room for controllable supervision at the plan level.<n>We introduce AuraGen, a controllable engine that synthesizes benign trajectories, injects category-labeled risks with difficulty, and filters outputs via an automated reward model.
  arXiv  Detail & Related papers  (2025-10-10T18:42:32Z)
• Safety Margins for Reinforcement Learning [53.10194953873209]
  We show how to leverage proxy criticality metrics to generate safety margins. We evaluate our approach on learned policies from APE-X and A3C within an Atari environment.
  arXiv  Detail & Related papers  (2023-07-25T16:49:54Z)
• Understanding the Vulnerability of Skeleton-based Human Activity Recognition via Black-box Attack [53.032801921915436]
  Human Activity Recognition (HAR) has been employed in a wide range of applications, e.g. self-driving cars. Recently, the robustness of skeleton-based HAR methods have been questioned due to their vulnerability to adversarial attacks. We show such threats exist, even when the attacker only has access to the input/output of the model. We propose the very first black-box adversarial attack approach in skeleton-based HAR called BASAR.
  arXiv  Detail & Related papers  (2022-11-21T09:51:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.