SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance

• URL: http://arxiv.org/abs/2608.09253v1
• Date: Mon, 10 Aug 2026 08:13:52 GMT
• Title: SkillSentry: Reliable Skill Execution for LLM Agents via Runtime Assurance
• Abstract summary: SkillSentry is a skill-oriented runtime assurance framework for representing runtime guidance for skill execution.<n>We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models.<n>Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.
• Score: 10.175883215226776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents are increasingly equipped with skills to perform complex tasks through multi-step reasoning and tool use. Although skills provide reusable procedural knowledge, agents may still execute them unreliably. Even when an agent has demonstrated the capability to complete tasks under the guidance of a skill, it may fail to do so consistently across similar tasks or repeated runs due to deviations from the skill procedure or incorrect execution of individual steps. Such instability limits the practical reliability of LLM agents. To address this problem, we propose SkillSentry, a skill-oriented runtime assurance framework built upon a new domain-specific language (DSL) for representing runtime guidance for skill execution. SkillSentry initializes the runtime guidance by combining a skill specification extracted from the corresponding skill document with execution experience mined from historical successful and failed traces. It then wraps around the agent execution loop to monitor and guide skill execution under the current guidance, while iteratively refining the guidance using newly collected traces. We evaluate SkillSentry on 15 skills across two LLM agents, each paired with two backbone models, i.e., Claude Code with Claude-Haiku-4.5 and Claude-Opus-4.6, and Codex with GPT-5.2 and GPT-5.4. Our results show that SkillSentry improves the task success rate of LLM agents by 24.1% across skills, on average, while exhibiting lower variability across repeated runs.

Related papers

• Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses? [38.776828282073865]
  We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure.<n> Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations.<n>A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered.
  arXiv  Detail & Related papers  (2026-08-05T13:29:16Z)
• Generative Skill Composition for LLM Agents [58.47191648555256]
  LLM agents benefit from skills for solving complex tasks.<n>As skill libraries grow, selecting an appropriate skill composition has emerged as a central bottleneck.<n>We propose SkillComposer, which instantiates structured skill composition as task-conditioned skill sequence prediction.
  arXiv  Detail & Related papers  (2026-06-30T17:53:09Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillComposer: Learning to Evolve Agent Skills for Specification and Generalization [99.18305770138677]
  Agent skills are reusable strategies that guide agent reasoning and action.<n>Current skill construction methods treat the problem as one-shot extraction.<n>We introduce SkillComposer, a framework that decomposes skill construction into three learnable operations.
  arXiv  Detail & Related papers  (2026-06-04T12:16:08Z)
• MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills? [36.863282825853645]
  MMG2Skill is a framework for converting in-the-wild guides into agent-executable skills.<n>MMG2Skill consistently outperforms vanilla baseline agents in every model-domain setting.
  arXiv  Detail & Related papers  (2026-06-01T09:50:40Z)
• SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision [41.562357872885826]
  We propose SkillRevise, an execution-grounded framework designed to iteratively refine agent skills.<n>SkillRevise diagnoses skill defects from execution evidence, retrieves relevant repair principles from a general memory, and applies execution-anchored edits.<n>It substantially outperforms one-shot baselines, improving the base agent's success rate on SkillsBench from 36.05% to 61.63%.
  arXiv  Detail & Related papers  (2026-05-31T10:19:13Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• SkillMaster: Toward Autonomous Skill Mastery in LLM Agents [27.651128308229378]
  We propose SkillMaster, a training framework that teaches agents to create new skills, refine existing skills, and select accumulated skills during task solving.<n>First, we train agents through trajectory-informed skill review, teaching agents to propose, update, or retain skills based on evidence from completed episodes.<n>Second, each candidate skill edit is designed to be evaluated by its counterfactual utility on related probe tasks, providing a direct learning signal for training skill-editing decisions.<n>Third, we introduce DualAdv-GRPO, which separately estimates advantages for task-solving actions and skill-editing decisions, stabilizing joint training across task solving and
  arXiv  Detail & Related papers  (2026-05-09T05:03:00Z)
• SkillOS: Learning Skill Curation for Self-Evolving Agents [67.94374107466957]
  We propose SkillOS, an experience-driven RL training recipe for learning skill curation in self-evolving agents.<n>SkillOS pairs a frozen agent executor that retrieves and applies skills with a trainable skill curator that updates an external SkillRepo from accumulated experience.<n>We show that SkillOS consistently outperforms memory-free and strong memory-based baselines in both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-05-07T17:31:50Z)
• How Well Do Agentic Skills Work in the Wild: Benchmarking LLM Skill Usage in Realistic Settings [29.3744517703302]
  We study the first comprehensive study of skill utility under progressively challenging realistic settings.<n>Our findings reveal that performance gains degrade consistently as settings become more realistic.<n>We show that query-specific refinement substantially recovers lost performance when the initial skills are of reasonable relevance and quality.
  arXiv  Detail & Related papers  (2026-04-06T00:10:30Z)
• EvoSkills: Self-Evolving Agent Skills via Co-Evolutionary Verification [85.3692584167951]
  Anthropic proposes the concept of skills for LLM agents to tackle multi-step professional tasks.<n>A tool is a single, self-contained function, whereas a skill is a structured bundle of interdependent multi-file artifacts.<n>EvoSkills is a self-evolving skills framework that enables agents to autonomously construct complex, multi-file skill packages.
  arXiv  Detail & Related papers  (2026-04-02T06:43:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.