Daily Paper Briefing

Abstract Overview

This paper introduces HarnessRisk, a lifecycle-oriented benchmark for agent harness safety that organizes risks into six operational phases: Harness Configuration, Capability Extension, Runtime Operation, State Persistence, Action Control, and Incident Recovery. The benchmark contains 128 sandboxed workflow cases in which a benign user task is paired with an adversarial instruction embedded in an untrusted artifact, and each run is scored on Utility, Attack Success Rate, Persistence, and Detection. The authors evaluate 14 model–harness configurations spanning three harnesses and six language models in isolated environments with mock services and no real external side effects. The study is motivated by the claim that agent safety depends jointly on the model and the harness, not on the model alone.

Novelty

The main novelty is the benchmark's lifecycle-based framing of agent harness safety, which evaluates six distinct operational responsibilities under a unified protocol rather than focusing mainly on runtime attacks. It is also distinctive in separately measuring task completion, attack success, durable compromise, and explicit risk detection across full agent trajectories.

Results

Across all 14 evaluated configurations, the authors find substantial safety failures despite generally high task utility, showing that successful task completion is not a reliable proxy for safe behavior. Harness choice materially changes outcomes for the same model, with reported attack success varying by more than fourfold in one example, and Harness Configuration is the highest-risk phase on all three harnesses. Detection is negatively associated with attack success, but high detection rates still do not guarantee remediation or prevention of unsafe actions.

Key Points

1. HarnessRisk benchmarks agent harness safety across six lifecycle phases using 128 sandboxed, multi-turn workflow cases.
2. The evaluation separates utility from safety by scoring each trajectory on Utility, Attack Success Rate, Persistence, and Detection.
3. Empirical results show that safety depends strongly on the deployed model–harness configuration, with high utility often coexisting with unsafe execution.

References

• arXiv: https://arxiv.org/abs/2608.17597v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2608.17597v1
• Hugging Face Papers: https://huggingface.co/papers/2608.17597