How Agent Skills Fail under Long Contexts: A White-Box Study in Code Auditing

• URL: http://arxiv.org/abs/2607.17937v1
• Date: Mon, 20 Jul 2026 13:34:53 GMT
• Title: How Agent Skills Fail under Long Contexts: A White-Box Study in Code Auditing
• Abstract summary: Agent Skills package procedural instructions and checks for use by general-purpose agents.<n>We study this problem in a production-derived, white-box code-audit workflow.<n>We classify where failures first become visible: lost requirements, editing drift, failed checking, or non-agent evaluator/runtime failures.
• Score: 1.5217170480235822
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Agent Skills package procedural instructions and checks for use by general-purpose agents, but loading a skill does not guarantee that every requirement remains active throughout a long tool-using trajectory. We study this problem in a production-derived, white-box code-audit workflow. Holding the task and 24 artifact checks fixed, we vary the surrounding context and classify where failures first become visible: lost requirements, editing drift, failed checking, or non-agent evaluator/runtime failures. Codex with gpt-5.4-mini passes 8/10 runs in a 10,991-character clean context but only 3/10 in both a 299,140-character relevant context and an equal-length irrelevant context. This 50-percentage-point difference is large but remains trend-level under two-sided Fisher tests (p = 0.0698). Requirement coverage nevertheless stays above 92% in both long conditions, showing that a few omissions can invalidate an otherwise complete artifact. A second task passes all clean and long runs, so the evidence does not support a universal context-length threshold. A detailed external checklist passes 10/10 runs, compared with 5/10 for a generic self-check (p = 0.0325). Coding-agent scaffolds may help by selecting a smaller working set, but they do not eliminate failures. We do not introduce context rot or a new general monitoring method; we provide a bounded failure classification and empirical case study for white-box code auditing.

Related papers

• Schwarz: Solver-Aware Agentic Program Verification [1.9215527229034466]
  Schwarz is an agentic verification harness that makes SMT-backed proof failure local, checkable, and repairable.<n>We implement Schwarz for C and Rust/Verus and evaluate it on 1,475 tasks.
  arXiv  Detail & Related papers  (2026-08-31T13:54:48Z)
• Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents [16.70932968825709]
  We introduce a knowledge-gated task-construction protocol that separates a task instruction from a compact artefact.<n>One frontier agent configuration achieves a 68.0% pass rate with the artefact and 0% without it.<n>We do not establish that the retained tasks improve post-training.
  arXiv  Detail & Related papers  (2026-08-31T06:39:27Z)
• APIFlow-Bench: Measuring Whether Agents Survive Long, Dependent API Workflows [23.621239974604205]
  APIFlow-Bench is a fully auditable benchmark for long-horizon dependent REST-API performance.<n>We generate synthetic API worlds forward, subtask by subtask.<n>We release all answer keys and 44,362 unredacted execution transcripts.
  arXiv  Detail & Related papers  (2026-08-29T08:12:11Z)
• Demystifying Agent Skills: Why They Work-Until They Don't [58.92875630998418]
  Existing evaluations largely measure whether skills improve aggregated task success, leaving a more fundamental question underexplored: emphtextbfWhen do skills help, why do they work, and where do they fail?<n>We isolate the effects of representation, outcome annotation, retrieval difficulty, and cross-framework of skills.<n>We consolidate these observations into a taxonomy of three high-level categories and twelve skill-use modes.
  arXiv  Detail & Related papers  (2026-08-14T07:26:38Z)
• Do Context Files Help Coding Agents? A Two-Agent Ablation Study on Real Repositories [0.0]
  Persistent context files (AGENTS.md, CLAUDE.md) are standard practice for guiding AI coding agents, yet evidence for their effectiveness is contradictory.<n>We present a controlled ablation of context-injection strategy across two frontier agents (Claude Code and Codex)
  arXiv  Detail & Related papers  (2026-07-28T11:07:53Z)
• Tencent WorkBuddy Bench: A Multi-Domain Coding-Agent Benchmark with Contamination-Resistant Task Construction [57.138342889101345]
  Tencent WorkBuddy Bench is a multi-domain evaluation suite for coding agents.<n>This report documents its construction methodology, scoring protocol, and a cross-model leaderboard.
  arXiv  Detail & Related papers  (2026-07-23T04:34:06Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• DeepSWE: Measuring Frontier Coding Agents on Original, Long-Horizon Engineering Tasks [1.7623000586936592]
  DeepSWE is a benchmark of 113 original, long-horizon software engineering tasks for evaluating coding agents.<n>Its tasks are written from scratch across 91 active open-source repositories and five languages and are never contributed back upstream.<n>Despite being about half the length of SWE-Bench Pro's prompts, DeepSWE's prompts describe tasks whose reference solutions touch 5.5x more code.
  arXiv  Detail & Related papers  (2026-07-08T21:45:34Z)
• Auditing Reward Hackability in Code RL Training Environments [0.0]
  We measure the rate at which code RL environments accept incorrect solutions as correct.<n>On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them.
  arXiv  Detail & Related papers  (2026-06-14T23:31:42Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• Getting Better at Working With You: Compiling User Corrections into Runtime Enforcement for Coding Agents [99.67681154381803]
  Test-time Rule Acquisition and Compiled Enforcement (TRACE) is a skill-layer pipeline for coding-agent runtimes.<n>Unlike runtime checks written ahead of time by developers, TRACE skills come from the user's own chat corrections.
  arXiv  Detail & Related papers  (2026-06-11T10:43:40Z)
• Verify-Gated Completion as Admission Control in a Governed Multi-Agent Runtime: A Bounded Architecture Case Study [0.6875312133832079]
  We study verify-gated completion as an admission-control pattern for governed multi-agent runtimes.<n>We show that under observed conditions, a read-only verify gate plus packetized admission records made completion decisions inspectable and fail-closed.
  arXiv  Detail & Related papers  (2026-05-18T07:52:13Z)
• Claw-Eval-Live: A Live Agent Benchmark for Evolving Real-World Workflows [67.92316850084575]
  We introduce Claw-Eval-Live, a live benchmark for workflow agents.<n>Each release is constructed from public workflow-demand signals.<n>Claw-Eval-Live records execution traces, audit logs, service state, and post-run workspace artifacts.
  arXiv  Detail & Related papers  (2026-04-30T17:23:19Z)
• ClawMark: A Living-World Benchmark for Multi-Turn, Multi-Day, Multimodal Coworker Agents [77.22389710754452]
  We introduce bench, a benchmark for coworker agents built around multi-turn multi-day tasks.<n>The current release contains 100 tasks across 13 professional scenarios, executed against five stateful sandboxed services.<n>The strongest model reaches 75.8 weighted score, but the best strict Task Success is only 20.0%, indicating that partial progress is common.
  arXiv  Detail & Related papers  (2026-04-26T16:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.