TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation

• URL: http://arxiv.org/abs/2608.17588v1
• Date: Tue, 18 Aug 2026 09:52:57 GMT
• Title: TRUSS: Towards Task-Reliable and User-Safe Automated Agent Skill Generation
• Abstract summary: TRUSS is an evidence guided framework for generating functionally effective and safety reliable Agent Skills.<n>We evaluate TRUSS on 168 SkillInject artifacts, 155 SkillSafetyBench cases, and all 187 tasks in SkillGenBench.<n>For Skill generation, TRUSS raises task effectiveness from 17.11% without Skills to 52.94%, while increasing the benchmark Security rate from 50.80% to 100.00%.
• Score: 13.298401993417611
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agent Skills package reusable natural language procedures with executable resources, enabling software agents to acquire task specific capabilities without model adaptation. Automatically generating such Skills can improve task performance, yet evaluating a candidate solely from its artifact or final task outcome leaves unresolved which actions the equipped agent will perform and which side effects those actions will produce. We present TRUSS, an evidence guided framework for generating functionally effective and safety reliable Agent Skills. TRUSS first inspects functional claims against source and domain evidence while evaluating the complete artifact under nine predefined safety properties. Candidates admitted by this static gate are loaded by a shadow agent inside a Controllable Execution Environment, where brokered tools expose requested actions to policy enforcement and record their results as provenance preserving execution traces. Functional failures and property violations are linked back to the responsible Skill content and used to guide iterative refinement. We evaluate TRUSS on 168 SkillInject artifacts, 155 SkillSafetyBench cases, and all 187 tasks in SkillGenBench. TRUSS achieves 100.00\% precision and recall in vulnerability detection. Repair reduces attack success from 38.71\% to 19.35\% with GPT 5.5 and from 46.45\% to 29.68\% with GPT 5.4, with zero attack regression. For Skill generation, TRUSS raises task effectiveness from 17.11\% without Skills to 52.94\%, while increasing the benchmark Security rate from 50.80\% to 100.00\%. These results show that execution evidence can expose behavioral failures missed by artifact inspection and can guide Skill generation toward jointly verified functional and safety outcomes.

Related papers

• BenchShield: Formal Model-Backed Instrumentation for Reward Integrity in LLM-Agent Evaluation Infrastructure [47.61632934139226]
  LM-agent benchmarks increasingly function as interactive evaluation infrastructure.<n>This interactivity makes evaluations vulnerable to reward hacking.<n>This paper presents BenchShield, a model-backed instrumentation layer for reward integrity.
  arXiv  Detail & Related papers  (2026-09-10T03:10:58Z)
• Defense-as-Skill: Evolving Runtime Guard Skill for Skill-Augmented Agents [30.77975669386977]
  We propose Defense-as-Skill, a defense paradigm that implements the runtime guard itself as an installable, inspectable, and editable skill.<n>Our guard, SkillSonar, checks sensitive actions against the user's task boundary, routing each action to an allow, replan, or confirmation decision.
  arXiv  Detail & Related papers  (2026-09-01T16:19:31Z)
• Task-Conditioned Least-Privilege Learning for Executable Terminal and MCP Agents [45.88028371034407]
  Tool-using large language-model agents can complete a task while exercising authority that the user did not grant or the task does not need.<n>We propose a framework where each action is audited before execution and again from observed effects along six dimensions of risk.<n>We find that after training using this framework on Qwen3.5-4B over 1,500 tasks, the selected seed reaches 98.48% safe success across 2,896 evaluation episodes.
  arXiv  Detail & Related papers  (2026-08-18T22:07:23Z)
• Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents [9.71657681051469]
  This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills.<n>We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions.<n>We propose research topics and tooling improvements for safer and more cost-aware skill reuse.
  arXiv  Detail & Related papers  (2026-08-12T10:15:19Z)
• ActBench: Self-Evolving Benchmark of Behavioral Safety in Cowork Agents [23.994328225666663]
  Cowork agents may complete benign tasks while disclosing protected data, manipulating unauthorized state, invocate unauthorized API.<n>We define behavioral safety and introduce ActBench, a benchmark that evaluates such behavior risk from execution trajectories rather than final responses.<n>ActBench contains 600 cases from 213 scenarios, spanning 15 risk behaviors, six execution spaces, and 48 web-service APIs.
  arXiv  Detail & Related papers  (2026-08-10T11:45:03Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• RedAct: Redacting Agent Capability Traces for Procedural Skill Protection [6.305785618804429]
  Execution traces contain rich procedural detail, including tool invocations, intermediate decisions, and error-recovery logic.<n>This detail can expose private procedural skills, allowing downstream methods to recover key formulas, thresholds, and strategies without access to model weights or skill files.<n>We introduce RedAct, a protected trace release framework that localizes protected key information, rewrites traces while preserving verifier-critical evidence, and embeds behavioral watermarks for downstream provenance analysis.
  arXiv  Detail & Related papers  (2026-06-09T12:57:50Z)
• SkillGuard: A Permission Framework for Agent Skills [14.787581333168859]
  We present SkillGuard, a skill-centric permission framework that treats skills as permission-bearing executable artifacts.<n>We evaluate SkillGuard on 315 real-world skills and SkillInject.
  arXiv  Detail & Related papers  (2026-06-02T02:01:53Z)
• SkillHarm: Lifecycle-Aware Skill-Based Attacks via Automated Construction [89.03272793385054]
  Agent skills occupy a privileged position in the agent workflow, rendering third-party skills a vulnerable attack surface.<n>We introduce SkillHarm, a benchmark of skill-based attacks across the skill-use lifecycle.<n>We show that current agents remain vulnerable with attack success rates up to 86.3% in FPP and 69.3% in SMP.
  arXiv  Detail & Related papers  (2026-06-01T17:45:39Z)
• On-Policy Self-Evolution via Failure Trajectories for Agentic Safety Alignment [54.30690671490447]
  Existing safety-alignment signals are largely response-level or off-policy.<n>FATE transforms verifier-scored failures into repair supervision without expert demonstrations.<n>FATE reduces attack success rate by 33.5%, harmful compliance by 82.6%, and improves external trajectory-safety diagnosis by 6.5%.
  arXiv  Detail & Related papers  (2026-05-12T09:56:28Z)
• SkillAttack: Automated Red Teaming of Agent Skills through Attack Path Refinement [66.44008181092832]
  LLM-based agent systems increasingly rely on agent skills sourced from open registries to extend their capabilities.<n>We introduce SkillAttack, a framework that verifies skill vulnerability exploitability through adversarial prompting.
  arXiv  Detail & Related papers  (2026-04-05T06:25:11Z)
• The Verifier Tax: Horizon Dependent Safety Success Tradeoffs in Tool Using LLM Agents [1.3299507495084417]
  We study how runtime enforcement against unsafe actions affects end-to-end task performance in multi-step tool using large language model (LLM) agents.<n>We identify model dependent interaction horizons (15 to 30 turns) and decompose outcomes into overall success rate (SR), safe success rate (SSR), and unsafe success rate (USR)
  arXiv  Detail & Related papers  (2026-03-18T16:16:34Z)
• Exploring Expert Failures Improves LLM Agent Tuning [74.0772570556016]
  We propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories.<n>EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance.
  arXiv  Detail & Related papers  (2025-04-17T17:53:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.