Human-Guided Harm Recovery for Computer Use Agents

• URL: http://arxiv.org/abs/2604.18847v1
• Date: Mon, 20 Apr 2026 21:12:40 GMT
• Title: Human-Guided Harm Recovery for Computer Use Agents
• Abstract summary: LM agents gain the ability to execute actions on real computer systems.<n>We need ways to not only prevent harmful actions at scale but also effectively remediate harm when prevention fails.<n>We formalize a solution to this neglected challenge in post-execution safeguards as harm recovery.
• Score: 7.834133575906748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As LM agents gain the ability to execute actions on real computer systems, we need ways to not only prevent harmful actions at scale but also effectively remediate harm when prevention fails. We formalize a solution to this neglected challenge in post-execution safeguards as harm recovery: the problem of optimally steering an agent from a harmful state back to a safe one in alignment with human preferences. We ground preference-aligned recovery through a formative user study that identifies valued recovery dimensions and produces a natural language rubric. Our dataset of 1,150 pairwise judgments reveals context-dependent shifts in attribute importance, such as preferences for pragmatic, targeted strategies over comprehensive long-term approaches. We operationalize these learned insights in a reward model, re-ranking multiple candidate recovery plans generated by an agent scaffold at test time. To evaluate recovery capabilities systematically, we introduce BackBench, a benchmark of 50 computer-use tasks that test an agent's ability to recover from harmful states. Human evaluation shows our reward model scaffold yields higher-quality recovery trajectories than base agents and rubric-based scaffolds. Together, these contributions lay the foundation for a new class of agent safety methods -- ones that confront harm not only by preventing it, but by navigating its aftermath with alignment and intent.

Related papers

• Diagnosis Before Recovery: Turning Agent Failures into Selective Self-Correction [22.654396704495934]
  Development-set failures can recover part of the missing diagnostic substrate by deciding which recovery interventions are admissible.<n>We propose DARC, a diagnosis-guided recovery harness that profiles task-family failure modes, prunes mismatched interventions from a shared recovery library, and freezes a verifier-selected success-cost policy for deployment.
  arXiv  Detail & Related papers  (2026-08-12T08:14:45Z)
• Oversight Has a Capacity: Calibrating Agent Guards to a Subjective, Fatiguing Human [0.0]
  As agents take irreversible actions, the standard safety pattern is a human-in-the-loop approval gate.<n>We argue the gate is the easy part; the hard part is the judgment - which actions to stop - which the field evaluates against two false assumptions.<n>Our contribution is an open-source agent-oversight system that operationalizes and measures them in the LLM-agent action-gating setting.
  arXiv  Detail & Related papers  (2026-06-08T01:52:22Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)
• Are Your Agents Upward Deceivers? [73.1073084327614]
  Large Language Model (LLM)-based agents are increasingly used as autonomous subordinates that carry out tasks for users.<n>This raises the question of whether they may also engage in deception, similar to how individuals in human organizations lie to superiors to create a good image or avoid punishment.<n>We observe and define agentic upward deception, a phenomenon in which an agent facing environmental constraints conceals its failure and performs actions that were not requested without reporting.
  arXiv  Detail & Related papers  (2025-12-04T14:47:05Z)
• Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Models [71.34520793462069]
  Unsupervised reinforcement learning (RL) aims at pre-training agents that can solve a wide range of downstream tasks in complex environments.<n>We introduce a novel algorithm regularizing unsupervised RL towards imitating trajectories from unlabeled behavior datasets.<n>We demonstrate the effectiveness of this new approach in a challenging humanoid control problem.
  arXiv  Detail & Related papers  (2025-04-15T10:41:11Z)
• LaMOuR: Leveraging Language Models for Out-of-Distribution Recovery in Reinforcement Learning [16.093659272414527]
  We introduce Language Models for Out-of-Distribution Recovery (LaMOuR), which enables recovery learning without relying on uncertainty estimation.<n>LaMOuR generates dense reward codes that guide the agent back to a state where it can successfully perform its original task.<n> Experimental results show that LaMOuR substantially enhances recovery efficiency across diverse locomotion tasks.
  arXiv  Detail & Related papers  (2025-03-21T13:20:39Z)
• Out-of-Distribution Recovery with Object-Centric Keypoint Inverse Policy for Visuomotor Imitation Learning [2.6696199945489534]
  We propose an object-centric recovery (OCR) framework to address the challenges of out-of-distribution (OOD) scenarios in visuomotor policy learning.<n>We show OCR's capacity to autonomously collect demonstrations for continual learning.
  arXiv  Detail & Related papers  (2024-11-05T17:41:14Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)
• MEReQ: Max-Ent Residual-Q Inverse RL for Sample-Efficient Alignment from Intervention [76.83428371942735]
  We introduce MEReQ (Maximum-Entropy Residual-Q Inverse Reinforcement Learning), designed for sample-efficient alignment from human intervention.<n>MereQ infers a residual reward function that captures the discrepancy between the human expert's and the prior policy's underlying reward functions.<n>It then employs Residual Q-Learning (RQL) to align the policy with human preferences using this residual reward function.
  arXiv  Detail & Related papers  (2024-06-24T01:51:09Z)
• A State Augmentation based approach to Reinforcement Learning from Human Preferences [20.13307800821161]
  Preference Based Reinforcement Learning attempts to solve the issue by utilizing binary feedbacks on queried trajectory pairs. We present a state augmentation technique that allows the agent's reward model to be robust.
  arXiv  Detail & Related papers  (2023-02-17T07:10:50Z)
• Transferable, Controllable, and Inconspicuous Adversarial Attacks on Person Re-identification With Deep Mis-Ranking [83.48804199140758]
  We propose a learning-to-mis-rank formulation to perturb the ranking of the system output. We also perform a back-box attack by developing a novel multi-stage network architecture. Our method can control the number of malicious pixels by using differentiable multi-shot sampling.
  arXiv  Detail & Related papers  (2020-04-08T18:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.