AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories

• URL: http://arxiv.org/abs/2609.06783v1
• Date: Sun, 06 Sep 2026 18:58:56 GMT
• Title: AURA-Eval: Evaluation Framework for Acting Under Risk Awareness in LLM Agent Trajectories
• Abstract summary: We introduce AURA-Eval, a framework combining controlled augmentation with granular diagnosis of behavior in tool-use trajectories.<n>Using 157 sourced trajectories, we generate 1,249 evaluation items and evaluate 20 frontier and open-weight models.<n>Our results show that LLM agents engage in unsafe behavior more often when no safe path exists.
• Score: 12.694014584212695
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM agents operate in workflows where unsafe actions can have real consequences. Existing safety evaluations often reduce behavior to a single score, obscuring risk recognition, pre-action detection, and safe task completion when a safe solution exists. We introduce AURA-Eval, a framework combining controlled augmentation with granular diagnosis of behavior in tool-use trajectories. Its pipeline identifies safety-critical decision points, generates controlled variations, and constructs counterparts differing in whether a request has a safe fulfillment path. Using 157 sourced trajectories, we generate 1,249 evaluation items and evaluate 20 frontier and open-weight models. We developed rubrics to classify risk detection, action strategy, and scenario-specific action safety. Our results show that LLM agents engage in unsafe behavior more often when no safe fulfillment path exists. In these cases, frontier proprietary models more often recognize risk and exhibit safer behavior by proposing alternatives, while evaluated open-weight models more often directly execute unsafe requests. Increasing impact or reducing opportunities for oversight before execution also exposes greater vulnerability across models.

Related papers

• Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification [32.337075755765056]
  Vera is an endto-end automated safety testing framework that instantiates software engineering testing principles for non-deterministic agents.<n>We release Vera, comprising 1600 executable safety cases spanning 124 risk categories across execution settings.<n>These results indicate that modular, executable testing infrastructure is essential for rigorous and maintainable safety evaluation of rapidly evolving agentic systems.
  arXiv  Detail & Related papers  (2026-07-02T07:08:26Z)
• TRACES: Proactive Safety Auditing for Multi-Turn LLM Agents via Trajectory-State Modeling [18.58206519209223]
  LLM agents increasingly operate through multi-turn tool use and environment interaction.<n>Post-hoc diagnosis frequently misses the chance to flag risks while they are unfolding.<n>We propose TRACES, a representation-based proactive auditor.
  arXiv  Detail & Related papers  (2026-05-26T21:11:02Z)
• Action-Conditioned Risk Gating for Safety-Critical Control under Partial Observability [79.08785366532287]
  We propose a lightweight risk-gated reinforcement learning approximation for risk-sensitive control under partial observability.<n>We evaluate the approach in two safety-critical partially observable domains: automated glucose regulation and safety-constrained navigation.
  arXiv  Detail & Related papers  (2026-05-14T01:23:09Z)
• Enhancing Agent Safety Judgment: Controlled Benchmark Rewriting and Analogical Reasoning for Deceptive Out-of-Distribution Scenarios [0.2578242050187029]
  ROME is a controlled benchmark-construction pipeline that rewrites known unsafe trajectories into deceptive evaluation instances.<n>ARISE is a retrieval-guided inference-time enhancement that retrieves ReAct-style analogical safety trajectories from an external analogical base and injects them as structured reasoning exemplars.
  arXiv  Detail & Related papers  (2026-05-05T00:21:00Z)
• HazardArena: Evaluating Semantic Safety in Vision-Language-Action Models [87.35765363039638]
  We introduce HazardArena, a benchmark designed to evaluate semantic safety in Vision-Language-Action models.<n>We find that VLA models trained exclusively on safe scenarios often fail to behave safely when evaluated in their corresponding unsafe counterparts.<n>We propose a training-free Safety Option Layer that constrains action execution using semantic attributes or a vision-language judge.
  arXiv  Detail & Related papers  (2026-04-14T08:32:02Z)
• CORA: Conformal Risk-Controlled Agents for Safeguarded Mobile GUI Automation [68.53387633351484]
  We propose CORA (COnformal Risk-controlled GUI Agent), a post-policy, pre-action safeguarding framework that provides statistical guarantees on harmful executed actions.<n> CORA reformulates safety as selective action execution: we train a Guardian model to estimate action-conditional risk for each proposed step.<n>To rigorously evaluate this paradigm, we introduce Phone-Harm, a new benchmark of mobile safety violations with step-level harm labels.
  arXiv  Detail & Related papers  (2026-04-10T09:41:21Z)
• Self-Guard: Defending Large Reasoning Models via enhanced self-reflection [54.775612141528164]
  Self-Guard is a lightweight safety defense framework for Large Reasoning Models.<n>It bridges the awareness-compliance gap, achieving robust safety performance without compromising model utility.<n>Self-Guard exhibits strong generalization across diverse unseen risks and varying model scales.
  arXiv  Detail & Related papers  (2026-01-31T13:06:11Z)
• RoboSafe: Safeguarding Embodied Agents via Executable Safety Logic [56.38397499463889]
  Embodied agents powered by vision-language models (VLMs) are increasingly capable of executing complex real-world tasks.<n>However, they remain vulnerable to hazardous instructions that may trigger unsafe behaviors.<n>We propose RoboSafe, a runtime safeguard for embodied agents through executable predicate-based safety logic.
  arXiv  Detail & Related papers  (2025-12-24T15:01:26Z)
• When Models Outthink Their Safety: Mitigating Self-Jailbreak in Large Reasoning Models with Chain-of-Guardrails [74.63933201261595]
  Large Reasoning Models (LRMs) demonstrate remarkable capabilities on complex reasoning tasks.<n>LRMs remain vulnerable to severe safety risks, including harmful content generation and jailbreak attacks.<n>We propose the Chain-of-Guardrail (CoG), a training framework that recomposes or backtracks unsafe reasoning steps.
  arXiv  Detail & Related papers  (2025-10-24T09:32:25Z)
• LM Agents May Fail to Act on Their Own Risk Knowledge [15.60032437959883]
  Language model (LM) agents pose a diverse array of potential, severe risks in safety-critical scenarios.<n>While they often answer "Yes" to queries like "Is executing sudo rm -rf /*' dangerous?", they will likely fail to identify such risks in instantiated trajectories.
  arXiv  Detail & Related papers  (2025-08-19T02:46:08Z)
• Shape it Up! Restoring LLM Safety during Finetuning [65.75757313781104]
  Finetuning large language models (LLMs) enables user-specific customization but introduces critical safety risks.<n>We propose dynamic safety shaping (DSS), a framework that uses fine-grained safety signals to reinforce learning from safe segments of a response while suppressing unsafe content.<n>We present STAR-DSS, guided by STAR scores, that robustly mitigates finetuning risks and delivers substantial safety improvements across diverse threats, datasets, and model families.
  arXiv  Detail & Related papers  (2025-05-22T18:05:16Z)
• Anomalous State Sequence Modeling to Enhance Safety in Reinforcement Learning [0.0]
  We propose a safe reinforcement learning (RL) approach that utilizes an anomalous state sequence to enhance RL safety. In experiments on multiple safety-critical environments including self-driving cars, our solution approach successfully learns safer policies.
  arXiv  Detail & Related papers  (2024-07-29T10:30:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.