Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents

• URL: http://arxiv.org/abs/2607.27484v1
• Date: Wed, 29 Jul 2026 21:59:20 GMT
• Title: Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents
• Abstract summary: We ask whether skill-augmented agents exhibit a textbfReasoning Backroom, a systematic gap between stated skill use and intervention-measured influence.<n>We introduce BACKTRACE, an evaluation framework that pairs each skill-conditioned answer with a matched no-skill counterfactual.<n>We instantiate the framework as BACKROOMBench, a verified testbed spanning controlled logic and competition mathematics.
• Score: 19.325317938976355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reusable skills are becoming a standard interface for extending language agents with task procedures. Yet evaluators usually infer skill use from visible reasoning or the agent's own attribution. These signals show what the agent appears to use, not whether the skill changed its decision. We ask whether skill-augmented agents exhibit a \textbf{Reasoning Backroom}, a systematic gap between stated skill use and intervention-measured influence. We introduce BACKTRACE, an evaluation framework that pairs each skill-conditioned answer with a matched no-skill counterfactual, intervenes on skill meaning, wording, identity, content, and assignment, and elicits attribution only after the answer is committed. We instantiate the framework as BACKROOMBench, a verified testbed spanning controlled logic and competition mathematics, multiple skill conditions, single-agent and multi-agent settings, and diverse model families. Our evaluation reveals a pervasive provenance failure. Across models and domains, stated skill use often remains stable while causal reliance and signed utility vary, producing both silent uptake and performative use. Behavioral effects follow procedural content more reliably than displayed skill identity, whereas stated attributions respond strongly to artifact availability. Observational detectors based on direct skill-use claims, text mentions, trace similarity, and an LLM judge do not identify which decisions actually depend on the skill. In multi-agent systems, skill influence can survive communication even after its source is lost, while no-skill teams still name skills and sources that were never supplied. These findings establish the Reasoning Backroom as a general AI provenance problem whose audit requires intervention.

Related papers

• SkillAlign: Aligning Skill Interfaces for LLM-based Agents [14.545296091572688]
  We argue that the same skill can help, distract, or mislead depending on how it is exposed.<n>We propose SkillAlign, a provider-agnostic framework that represents candidate skills as multiview procedural cards.<n>Our results suggest that skill-augmented agents should optimize not only which skills to use, but also how those skills are presented.
  arXiv  Detail & Related papers  (2026-09-07T09:11:03Z)
• Coalition-Aware Skill Reliability for Self-Evolving Agents [49.43736235959317]
  We investigate whether accumulated skills in an agent's skill bank actually make positive mechanistic contributions.<n>We show that coalition pollution, where bank-level gains conceal negative coalition-level skill contributions, and cross-domain utility reversal, where source-beneficial skills reverse their effects after transfer.<n>These findings motivate two reliability interventions: coalition-aware skill selection during skill accumulation and label-free skill masking after transfer.
  arXiv  Detail & Related papers  (2026-08-23T21:35:22Z)
• SkillCommit: Evolving Agent Skills through Behaviorally Validated Scope Expansion [15.498568310060229]
  We propose SkillCommit, an online skill evolution framework that transforms experience into a hierarchical library of reusable skills.<n>As related skills accumulate, SkillCommit abstracts those sharing a common behavioral mechanism into higher-level skills.<n> Experiments on RuleArena, OpenExempt and KOR-Bench demonstrate that SkillCommit consistently improves agent performance across diverse domains.
  arXiv  Detail & Related papers  (2026-08-15T11:03:07Z)
• Skill-Use: Can LLMs Actually Use Skills in Agentic Harnesses? [38.776828282073865]
  We introduce Skill-Use, a benchmark that evaluates skill use under progressive disclosure.<n> Trigger measures whether the agent invokes the relevant skill, Compliance measures how faithfully it follows the prescribed procedure, and Boundary measures whether it avoids forbidden operations.<n>A Skill-Use (SU) score combines the three and credits execution only after the skill is triggered.
  arXiv  Detail & Related papers  (2026-08-05T13:29:16Z)
• SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment [10.850946672552277]
  We introduce SkillAudit, an end-to-end framework for skill-centered assessment.<n>It generates a comprehensive, multi-dimensional evaluation report spanning utility, efficiency/cost, and safety.<n>After scanning top-ranked real-world skill packages, we found that over 7% of skills are at risky status.
  arXiv  Detail & Related papers  (2026-06-21T17:40:05Z)
• SkillAudit: Ground-Truth-Free Skill Evolution via Paired Trajectory Auditing [81.51044612408793]
  SkillAudit is a framework for evolving agent skills without ground-truth feedback.<n>To turn behavioral differences into edit guidance, SkillAudit uses Process-Aligned Contrastive Evaluation.<n>It routes edits through two pipelines: Refine removes noisy or irrelevant guidance from broadly useful skills, while Repair replaces passages that conflict with the task.
  arXiv  Detail & Related papers  (2026-06-12T08:20:09Z)
• SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior [23.11308948097101]
  Current benchmarks rarely distinguish what a Skill says from how it is organized.<n>We study this distinction through Progressive Disclosure, where a concise root file points agents to supporting resources on demand.<n>We present SkillJuror, a framework for evaluating Skill writing paradigms.
  arXiv  Detail & Related papers  (2026-06-10T01:11:50Z)
• Co-Evolving Skill Generation and Policy Optimization [35.41582114275514]
  Existing methods typically use strong language models to analyze trajectories, generate skills, and update a retrievable skill bank during online training.<n>We propose an online reinforcement learning framework for pre-storage skill validation.
  arXiv  Detail & Related papers  (2026-06-07T17:55:55Z)
• SkillHone: A Harness for Continual Agent Skill Evolution Through Persistent Decision History [4.682850246574405]
  SkillHone is a harness for continual agent skill evolution grounded in persistent decision history.<n> SkillHone pairs skill revisions with evaluation-side evidence that supplies practice feedback.<n>We evaluate SkillHone on deep-research benchmarks in a raw open-web setting, where agents are not given an integrated search stack.
  arXiv  Detail & Related papers  (2026-06-07T15:21:08Z)
• COLLEAGUE.SKILL: Automated AI Skill Generation via Expert Knowledge Distillation [58.84646485020439]
  We present an automated trace-to-skill distillation system for generating person-grounded AI skills.<n> COLLEAGUE.SKILL produces a versioned skill package with two tracks: a capability track for practices, mental models, and decisions, and a bounded behavior track for communication style, interaction rules, and correction history.
  arXiv  Detail & Related papers  (2026-05-29T12:59:08Z)
• Under the Hood of SKILL.md: Semantic Supply-chain Attacks on AI Agent Skill Registry [49.83108591873481]
  We study SKILL.md - only attacks across three registry-facing stages of the Agent Skill lifecycle.<n>We show that SKILL.md is not passive documentation but operational text that shapes which third-party capabilities agents find, trust, and use.
  arXiv  Detail & Related papers  (2026-05-12T02:11:54Z)
• SkillGen: Verified Inference-Time Agent Skill Synthesis [60.927977774369516]
  SkillGen is a framework that synthesizes a single auditable skill from trajectories generated by a base agent.<n>It identifies reusable success patterns, recurring failure modes, and behaviors that appear in nearby successes but are missing from failures.<n>A key novelty in SkillGen is that we model agent skills as interventions to empirically verify the net effect of skills on the overall performance.
  arXiv  Detail & Related papers  (2026-05-09T19:24:11Z)
• From Context to Skills: Can Language Models Learn from Context Skillfully? [56.5857437218136]
  Ctx2Skill is a self-evolving framework that autonomously discovers, refines, and selects context-specific skills.<n>It can be plugged into any language model to obtain better context learning capability.
  arXiv  Detail & Related papers  (2026-04-30T09:53:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.