SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment

• URL: http://arxiv.org/abs/2609.02786v1
• Date: Wed, 02 Sep 2026 16:19:54 GMT
• Title: SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment
• Abstract summary: We propose SafeEvolve, an experience-driven framework for agent safety alignment.<n>We leverage safety experience from completed on-policy trajectories to drive a continual loop of harness-policy co-evolution.<n>For Qwen3.5-4B, SafeEvolve achieves a $3times$ ASR reduction on AgentDojo while improving benign utility from 59.79% to 61.86%.
• Score: 51.263370090136696
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The performance of LLM-based agents is jointly shaped by the base model and the harness used when interacting with the environment. This exposes them to safety risks in both harmful final responses and multi-step execution trajectories. Existing safety alignment mechanisms often rely on either external harness updates or policy optimization, yet applying either paradigm in isolation fails to bridge runtime control with intrinsic safety. We propose SafeEvolve, an experience-driven self-evolving framework for agent safety alignment. SafeEvolve leverages safety experience from completed on-policy trajectories to drive a continual loop of harness-policy co-evolution. On the harness side, SafeEvolve converts trajectory-level safety evidence into bounded, component-level updates across safety prompt and hierarchical skills, yielding auditable and reversible harness artifacts. On the policy side, SafeEvolve follows a two-stage SFT-RL paradigm, where harness-use SFT bootstraps the policy to actively leverage evolved harness artifacts, and harness-augmented RL further shapes autonomous safety behaviors during multi-step exploration via verifier-decomposed rewards. Through harness-policy co-evolution, SafeEvolve converts safety experience into an evolved runtime harness and improved policy behavior. Experiments on agentic safety benchmarks show that SafeEvolve achieves a stronger safety-utility tradeoff than existing baselines. For Qwen3.5-4B, SafeEvolve achieves a $3\times$ ASR reduction on AgentDojo while improving benign utility from 59.79% to 61.86%.

Related papers

• SHE: Trajectory-driven Safety Harness Evolution for LLM Agents [63.74895225415523]
  Existing safety mechanisms treat the harness as a fixed deployment artifact, limiting their ability to evolve with emerging risks.<n>We propose Safety Harness Evolution, a framework that learns evolving safe boundaries from rollout trajectories.<n>Experiments on Agent-SafetyBench demonstrate that SHE effectively enhances safety through harness evolution, achieving a 3.1x ASR reduction compared with static SafeHarness.
  arXiv  Detail & Related papers  (2026-08-10T17:35:08Z)
• RUBAS: Rubric-Based Reinforcement Learning for Agent Safety [93.80584563695646]
  RUBAS is a rubric-based reinforcement learning framework for agent safety.<n>It decomposes agent behavior into four dimensions: tool-use safety, argument safety, response safety, and helpfulness.<n>Experiments show that RUBAS improves safety over standard alignment baselines, reduces tool-grounded hallucinations, and maintains competitive utility.
  arXiv  Detail & Related papers  (2026-06-02T09:02:14Z)
• On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment [54.30690671490447]
  Existing safety-alignment signals are largely response-level or off-policy.<n>FATE transforms verifier-scored failures into repair supervision without expert demonstrations.<n>FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%.
  arXiv  Detail & Related papers  (2026-05-12T09:56:28Z)
• Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• Safety Modulation: Enhancing Safety in Reinforcement Learning through Cost-Modulated Rewards [23.15178050525514]
  Safe Reinforcement Learning (Safe RL) aims to train an RL agent to maximize its performance in real-world environments while adhering to safety constraints.<n>We propose a novel safe RL approach called Safety Modulated Policy Optimization (SMPO), which enables safe policy function learning.
  arXiv  Detail & Related papers  (2025-04-03T21:35:22Z)
• Safe Reinforcement Learning with Learned Non-Markovian Safety Constraints [15.904640266226023]
  We design a safety model that performs credit assignment to assess contributions of partial state-action trajectories on safety. We derive an effective algorithm for optimizing a safe policy using the learned safety model. We devise a method to dynamically adapt the tradeoff coefficient between safety reward and safety compliance.
  arXiv  Detail & Related papers  (2024-05-05T17:27:22Z)
• Safeguarded Progress in Reinforcement Learning: Safe Bayesian Exploration for Control Policy Synthesis [63.532413807686524]
  This paper addresses the problem of maintaining safety during training in Reinforcement Learning (RL) We propose a new architecture that handles the trade-off between efficient progress and safety during exploration.
  arXiv  Detail & Related papers  (2023-12-18T16:09:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.