ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation

• URL: http://arxiv.org/abs/2604.23853v1
• Date: Sun, 26 Apr 2026 19:44:10 GMT
• Title: ClawTrace: Cost-Aware Tracing for LLM Agent Skill Distillation
• Abstract summary: We introduce ClawTrace, an agent tracing platform that records every LLM call, tool use, and sub-agent spawn during an agent session.<n>CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches.<n>We show that both cost attribution and prune patches independently reduce quality regressions.
• Score: 9.992155237142306
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Skill-distillation pipelines learn reusable rules from LLM agent trajectories, but they lack a key signal: how much each step costs. Without per-step cost, a pipeline cannot distinguish adding a missing step to fix a bug from removing an expensive step that never affected the outcome. We introduce ClawTrace, an agent tracing platform that records every LLM call, tool use, and sub-agent spawn during an agent session and compiles each session into a TraceCard: a compact YAML summary with per-step USD cost, token counts, and redundancy flags. Built on ClawTrace, CostCraft is a distillation pipeline that reads TraceCards and produces three types of skill patches. Preserve patches keep behaviors that led to success. Prune patches remove expensive steps that did not matter, each backed by a counterfactual argument against a named high-cost step. Repair patches fix failures grounded in oracle evidence. Ablations on 30 held-out SpreadsheetBench tasks show that both cost attribution and prune patches independently reduce quality regressions. When the same skill is applied to 30 unrelated SkillsBench tasks, an unexpected asymmetry emerges: prune rules transferred across benchmarks and cut median cost by 32%, while preserve rules, trained on benchmark-specific conventions, caused regressions on new task types. We release ClawTrace and TraceCards as open infrastructure for cost-aware agent research.

Related papers

• Forgetting Without Restarting: Execution-State Unlearning for Stateful LLM Agents [12.985773309621964]
  Provenance-Guided Selective Replay attains bound as a cross-layer contract spanning prompt, compressed memory, and cache.<n>We audit with elicitation, cache, and string-free behavioral tests across three agent suites, nine baselines, and three model families.
  arXiv  Detail & Related papers  (2026-09-04T08:31:40Z)
• Codebook Agent: Amortized Topology Design for LLM Multi-Agent Systems [67.04448659688579]
  We develop a query-independent 16-entry codebook that ranks the top decoded candidates in a single batched forward pass.<n>With no iterative search and no message passing at test time, Codebook Agent is the most accurate method on all six benchmarks.
  arXiv  Detail & Related papers  (2026-09-02T08:10:22Z)
• When Not to Imitate: Boundary-Aware Skill Memory for Reliable Tool-Use LLM Agents [62.10623192614306]
  Extracting skills from past successes is critical for the efficient evolution of Large Language Model (LLM) agents.<n>We propose textbfBoundary-Aware Skill Memory (BASM), which augments each skill with explicit boundary fields.<n>BASM consistently outperforms success-distilled skill-memory baselines.
  arXiv  Detail & Related papers  (2026-08-23T10:16:06Z)
• TraceCompiler: Skill-Guided Mining and Compilation of LLM Agent Traces into Mostly Deterministic Workflows [2.435006380732194]
  We present TraceCompiler, a skill-guided system that compiles noisy traces into trajectories.<n>Every hard edge carries an auditable evidence, and ambiguous relations are marked suspected.<n>We measure call reduction but not offline compilation cost, so we claim no net efficiency result.
  arXiv  Detail & Related papers  (2026-08-03T01:05:06Z)
• Enhancing Software Engineering Through Closed-Loop Memory Optimization [50.418699819003486]
  We introduce ours, a closed-loop framework for memory augmentation in software engineering (SE) agents.<n>ours grounds memory utility in textitvalidated downstream impact, establishing utility as a task-agnostic textbf benchmark and an annotation-free textbfoptimization signal.<n>Results demonstrate that ours consistently improves SE agents across settings, achieving absolute gains of up to $uparrow5.25%$ in success rate and $uparrow4.63%$ in resolve efficiency.
  arXiv  Detail & Related papers  (2026-06-04T03:17:21Z)
• From Patches to Trajectories: Privileged Process Supervision for Software-Engineering Agents [56.31499185764872]
  Supervised fine-tuning (SFT) on long teacher trajectories is the dominant way to instill investigation and reasoning in open software-engineering (SWE) agents.<n>We propose Patches-to-Trajectories (P2T), which uses $pstar$ as privileged information during curation and formulates trajectory construction as bi-objective optimization over per-step effectiveness and trajectory length.
  arXiv  Detail & Related papers  (2026-05-21T04:54:55Z)
• Correction and Corruption: A Two-Rate View of Error Flow in LLM Protocols [51.56484100374058]
  We propose a paired-outcome measurement interface for auditing a single protocol step on exact-match tasks.<n>For each instance, the interface records a baseline correctness bit and a post-step correctness bit.<n>These rates predict accuracy changes and define a reusable empirical interface testable across seeds, mixtures, and pipelines.
  arXiv  Detail & Related papers  (2026-04-20T13:25:40Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• Runtime Execution Traces Guided Automated Program Repair with Multi-Agent Debate [8.424102114588559]
  Automated Program Repair (APR) struggles with complex logic errors and silent failures.<n>Current LLM-based APR methods are mostly static, relying on source code and basic test outputs.<n>We propose TraceRepair, a multi-agent framework that leverages runtime facts as shared constraints for patch validation.
  arXiv  Detail & Related papers  (2026-04-03T02:23:25Z)
• EcoScratch: Cost-Effective Multimodal Repair for Scratch Using Execution Feedback [3.6908036186618314]
  We present EcoScratch, a repair pipeline that uses lightweight runtime signals to decide whether the next attempt stays text-only or escalates to a multimodal prompting.<n>We evaluate 12 models on 100 executable Scratch repair projects under four controller settings, yielding 4800 repair trajectories.<n>It reaches the highest generation success (30.3%) while using less average cost and local-runtime energy than the two non-adaptive multimodal baselines under the same bounded trajectory budget; text-only lowest-cost floor.
  arXiv  Detail & Related papers  (2026-03-31T11:45:36Z)
• Large Language Models as Optimization Controllers: Adaptive Continuation for SIMP Topology Optimization [4.365822392824561]
  We present a framework in which a large language model acts as an online controller for SIMP optimization.<n>A hard grayness gate prevents premature binarization, and a meta-optimization loop uses a second pass to tune the agent's call frequency.<n>The agent achieves the final compliance on every benchmark: $-5.7%$ to $-1% relative to the fixed baseline, with all solutions fully binary.
  arXiv  Detail & Related papers  (2026-03-26T07:14:31Z)
• Agents Learn Their Runtime: Interpreter Persistence as Training-Time Semantics [4.774525456207306]
  Tool-augmented LLMs are increasingly deployed as agents that interleave natural-language reasoning with executable Python actions.<n>We ask whether interpreter persistence is merely a runtime scaffold, or a property of the training data that shapes how agents learn to use the interpreter.<n>We introduce Opaque Knapsack, a procedurally generated family of partially observable optimization tasks designed to prevent one-shot solutions.
  arXiv  Detail & Related papers  (2026-03-01T18:08:02Z)
• Beyond Max Tokens: Stealthy Resource Amplification via Tool Calling Chains in LLM Agents [31.789859492703016]
  The agent-tool communication loop is a critical attack surface in Large Language Model (LLM) agents.<n>Existing Denial-of-Service (DoS) attacks are ineffective for this new paradigm.<n>We introduce a stealthy, multi-turn economic DoS attack that operates at the tool layer under the guise of a correctly completed task.
  arXiv  Detail & Related papers  (2026-01-16T02:47:45Z)
• SABER: Small Actions, Big Errors -- Safeguarding Mutating Steps in LLM Agents [52.20768003832476]
  We analyze execution traces on $$-Bench (Airline/Retail) and SWE-Bench Verified.<n>We formalize emphdecisive deviations, earliest action, level divergences that flip success to failure.<n>We introduce cm, a model-agnostic, gradient-free, test-time safeguard.
  arXiv  Detail & Related papers  (2025-11-26T01:28:22Z)
• Runaway is Ashamed, But Helpful: On the Early-Exit Behavior of Large Language Model-based Agents in Embodied Environments [54.67512489842682]
  Large language models (LLMs) have demonstrated strong planning and decision-making capabilities in complex embodied environments.<n>We take a first step toward exploring the early-exit behavior for LLM-based agents.
  arXiv  Detail & Related papers  (2025-05-23T08:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.