Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents

• URL: http://arxiv.org/abs/2608.08793v1
• Date: Sun, 09 Aug 2026 16:18:13 GMT
• Title: Evidence-Calibrated Runtime Reconstruction for Agent Skills Across Heterogeneous Coding Agents
• Abstract summary: We present Skill Intelligence, a passive runtime-intelligence system that reconstructs supported Skill-lifecycle stages across heterogeneous harnesses.<n>We show that event presence is not boundary fidelity, composite exact scores mask distinct errors, and model explanations must not overwrite deterministic facts.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Agent Skills package reusable instructions and assets for tool-using language-model agents. Progressive loading creates failure boundaries poorly represented by session-, model-, or tool-centric traces: a Skill can be discovered but not activated, activated without instructions, or appear successful without an independently verified outcome. We present Skill Runtime Intelligence, a passive runtime-intelligence system that reconstructs supported Skill-lifecycle stages across heterogeneous harnesses while preserving unsupported stages as unknown. Its Run Panorama separates immutable events, deterministic relations, inferred diagnoses, and controlled outcomes with four evidence grades; optional trace import and OTLP/HTTP export support existing observability deployments. Across six frozen repository profiles, three coding agents, and seven clean or fault-injected conditions, all 126 executions preserve source worktrees and each correlates to exactly one source session. Yet adapters expose three distinct semantics: no Skill runs; complete runs but no failure-like events; or failure-like events in every operational-failure and clean session. In a seven-template diagnostic study, semantic aliases and Panorama localize the same six non-clean boundaries but differ in exact/status behavior; both Raw views emit a failure status on all 18 clean cases, while Panorama emits none. A known-rule graph conforms to 126/126 frozen contracts, whereas a second model completes only 228/378 calls. These observations motivate executable adapter qualification and show that event presence is not boundary fidelity, composite exact scores mask distinct errors, and model explanations must not overwrite deterministic facts.

Related papers

• Agent Mesh: Reliability Primitives for Non-Idempotent Agent Delegation - Identity Adequacy and Evidence Adequacy [0.0]
  We report a failure study of a production agentic software-delivery platform over 147 numbered incidents spanning 81 runs.<n>We quantify the consequences: a loop of fifty-four consecutive successful tool calls no error-rate breaker could see.<n>We derive seven reliability primitives whose enforcement unit is the delegation rather than the message.
  arXiv  Detail & Related papers  (2026-08-26T15:38:21Z)
• Beyond Multimodal Alignment: Certifying Physical Language through Response Substitution and Ordered Execution [18.546826658944806]
  We introduce an operational capability hierarchy and the Disjoint-Bridge Operator-Substitution Certificate.<n>On Cluster Haptic, audio and acceleration representations of the same unseen surface are 4.5x closer in response space than wrong-surface pairings.<n>We then test ordered execution in a controlled elastoplastic system with complementary modality blind spots.
  arXiv  Detail & Related papers  (2026-08-19T23:04:53Z)
• ScrambleToolBench: Agents Search Exhaustively Even When Their Own Map Points to the Next Step [74.83535434786145]
  We introduce ScrambleToolBench, an interactive terminal benchmark designed to isolate behavioral reasoning.<n>The benchmark requires agents to uncover hidden tool behaviors entirely through trial-and-error interaction.<n>Our evaluation of state-of-the-art language models reveals that successful initial discovery does not translate into robust adaptation.
  arXiv  Detail & Related papers  (2026-08-03T15:07:46Z)
• TRACE-CTI: Auditable Post-Extraction Governance of TTP Claims with Knowledge Graphs [4.960649631468518]
  TRACE-CTI is a post-extraction claim-governance framework.<n>It aggregates run-level Predictions into configuration-level GraphAssertions.<n>It materializes setup-deduplicated corroboration as ConsensusAssertions.
  arXiv  Detail & Related papers  (2026-07-27T15:33:18Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• Fantastic Adaptive Taxonomies and How to Use Them [100.20614173157708]
  AdaMAST converts execution traces into a compact, evidence-grounded failure taxonomy.<n>No code is hand-authored, no trace human-annotated.<n>In agent-system search, taxonomy-coded diagnoses of failed candidates outperform free-form.
  arXiv  Detail & Related papers  (2026-07-17T17:41:16Z)
• AgentAbstain: Do LLM Agents Know When Not to Act? [19.223843408018997]
  Existing evaluations mostly focus on task success rather than whether agents know when to abstain.<n>We present the first systematic evaluation framework for agentic abstention: the calibrated ability of tool-using LLM agents to recognize when not to act.<n>AgentAbstain is a paired-task benchmark built on an agent-native taxonomy of 8 abstention scenarios across pre-execution reasoning and runtime discovery.
  arXiv  Detail & Related papers  (2026-07-11T00:57:36Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• AgentAtlas: Beyond Outcome Leaderboards for LLM Agents [0.025718125188898048]
  AgentAtlas reframes agent evaluation as a diagnostic vocabulary and audit protocol.<n>The paper contributes: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a trajectory-failure vocabulary with primary error source and downstream impact; and (iv) an illustrative protocol study on a synthetic 1,342-item set evaluated with eight models.
  arXiv  Detail & Related papers  (2026-05-19T22:05:12Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• Claw-Eval: Toward Trustworthy Evaluation of Autonomous Agents [66.97968363332465]
  We introduce Claw-Eval, an end-to-end evaluation suite addressing all three gaps in agent benchmarks.<n>Claw-Eval comprises 300 human-verified tasks spanning 9 categories across three groups.<n>Every agent action is recorded through three independent evidence channels.
  arXiv  Detail & Related papers  (2026-04-07T17:43:18Z)
• Silent Commitment Failure in Instruction-Tuned Language Models: Evidence of Governability Divergence Across Architectures [0.0]
  We introduce governability -- the degree to which a model's errors are detectable before output commitment and correctable once detected.<n>We show benchmark accuracy does not predict governability, correction capacity varies independently of detection, and identical governance scaffolds produce opposite effects across models.<n>We propose a Detection and Correction Matrix classifying model-task combinations into four regimes: Governable, Monitor Only, Steer Blind, and Ungovernable.
  arXiv  Detail & Related papers  (2026-03-22T21:50:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.