SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing
Abstract Overview
SkillAudit is a framework for improving agent skills without using hidden tests, reference solutions, rewards, or other external ground-truth signals during optimization. Its core mechanism is paired trajectory auditing, where the same task is executed with and without the candidate skill so that the system can isolate the skill’s behavioral effect. These trajectory differences are interpreted by Process-Aligned Contrastive Evaluation (PACE), which produces segment-anchored diagnostic signals tied to specific parts of the skill document, while a fixed Anchor Verifier compiled from task specifications guards against harmful regressions. The method also routes edits through two pipelines: Refine for removing noisy guidance from broadly useful skills and Repair for replacing task-conflicting guidance.
Novelty
The paper’s main distinction is a ground-truth-free skill evolution loop that derives optimization signals solely from paired with-skill versus without-skill executions, rather than from oracle feedback or held-out evaluation signals. It also combines segment-level contrastive diagnosis (PACE) with a fixed structural verifier and dual Refine/Repair editing strategies.
Results
On 89 containerized tasks spanning 8 professional domains, SkillAudit reaches a 73.9% average task reward, compared with 40.9% for no-skill and 56.7% for the static benchmark skill. The analysis further shows that the method generally preserves already strong skills, partially recovers weak ones, and works best when task correctness leaves observable structural traces that the auditing process can detect.
Key Points
- SkillAudit improves skills by contrasting paired execution trajectories with and without the skill, avoiding dependence on hidden tests, reference answers, or reward signals during evolution.
- PACE provides localized edit guidance by mapping trajectory divergences to specific skill passages, while the Anchor Verifier supplies deterministic structural checks that can veto regressions.
- Empirically, the method improves over the static skill baseline in seven of eight domains, and the paper identifies observability of correctness signals—not domain alone—as a key factor governing when ground-truth-free skill evolution succeeds.