TraceLab: Characterizing Coding Agent Workloads for LLM Serving

• URL: http://arxiv.org/abs/2606.30560v2
• Date: Tue, 30 Jun 2026 15:39:15 GMT
• Title: TraceLab: Characterizing Coding Agent Workloads for LLM Serving
• Abstract summary: We release a trace of roughly 4,300 coding-agent sessions, containing about 350,000 LLM steps and 430,000 tool calls.<n>Our analysis shows that coding-agent workloads feature long autonomous loops, long contexts with short outputs, diverse and heavily-tailed tool calls, and high but imperfect prefix cache hit rates.<n>These findings point to concrete opportunities for optimizing serving, including lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and improved KV-cache management around human-paced gaps.
• Score: 16.402747744601612
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Coding agents are rapidly becoming a major application of agentic LLMs, but serving them efficiently remains challenging. Progress on this challenge requires understanding real workload patterns, yet the data needed for such analysis is largely absent. Existing public traces and benchmarks do not capture real, day-to-day coding-agent usage across multiple agents and model families for serving-system analysis. To help fill this gap, we collect and release a trace of roughly 4,300 coding-agent sessions, containing about 350,000 LLM steps and 430,000 tool calls from our own day-to-day use of Claude Code and Codex. Our analysis shows that coding-agent workloads feature long autonomous loops, long contexts with short outputs, diverse and heavily-tailed tool calls, and high but imperfect prefix cache hit rates. These findings point to concrete opportunities for optimizing serving, including lower-overhead tool calling, append-length-aware prefill, semantic-aware tool-latency prediction, and improved KV-cache management around human-paced gaps. We release the dataset, trace collection pipeline, and analysis code at https://github.com/uw-syfi/TraceLab.git the project website is https://tracelab.cs.washington.edu.

Related papers

• Agentic Coding in the Wild: Characterizing GitHub Copilot Traces at Production Scale [6.287658270510339]
  AI coding agents like GitHub Copilot, Claude Code, and Codex interleave multi-step LLM inference with tool execution.<n>We present the first production-scale characterization of this workload using sampled GitHub Copilot traces from June 2026.
  arXiv  Detail & Related papers  (2026-07-30T20:51:51Z)
• Towards LLM Accelerated Rapid Reviews for Software Tool Discovery -- Case for Log Anomaly Detection [46.43844768475097]
  We propose a pipeline to identify relevant studies with LLM screening, extract the tools presented in them, and run them with LLM-based coding agent.<n>Our approach would require roughly 4 hours of human effort, of which 3 hours were manual PDF downloading, and 12 hours of LLM running time.<n>Because practitioner-built tools often lack academic papers, in the future we aim to expand our analysis to tool-hosting platforms such as GitHub and PyPI.
  arXiv  Detail & Related papers  (2026-06-15T15:17:52Z)
• Can LLM Coding Agents Reason About Time Series? [0.19116784879310025]
  We show that agents with code access can outperform models processing raw data by up to 10%.<n>Even the best performing agent still answers about 22-34% of the questions incorrectly.<n>Our analysis reveals that coding agents can select appropriate statistical tests, but often miss important nuances.
  arXiv  Detail & Related papers  (2026-06-15T10:49:28Z)
• GTA: Generating Long-Horizon Tasks for Web Agents at Scale [82.43869456830664]
  We introduce a scalable framework, GTA, that integrates crawling, retrieval-based seeding, in-context generation, and automated quality control.<n>We instantiate the pipeline on over 50 websites covering e-commerce, government, forums, and news, with multilingual and multi-hop coverage.<n>Our contributions are three-fold: (i) formalizing multi-hop web-agent task generation, (ii) proposing an efficient and validated pipeline for automatic data creation, and (iii) releasing a dynamic benchmark with reproducible evaluation.
  arXiv  Detail & Related papers  (2026-05-28T01:05:50Z)
• MatClaw: An Autonomous Code-First LLM Agent for End-to-End Materials Exploration [0.0]
  We present MatClaw, a code-first agent that writes and executes Python directly.<n>We show that MatClaw handles code generation reliably but struggles with tacit domain knowledge.<n>Our results demonstrate that the gap between guided and fully autonomous computational materials research is narrower than ever before.
  arXiv  Detail & Related papers  (2026-04-03T03:32:15Z)
• TL-Training: A Task-Feature-Based Framework for Training Large Language Models in Tool Use [72.32614703504122]
  Large language models (LLMs) achieve remarkable advancements by leveraging tools to interact with environments.<n>Standard supervised fine-tuning approach, which relies on large-scale datasets, often overlooks task-specific characteristics in tool use.<n>We propose TL-Training, a task-feature-based framework that mitigates the effects of suboptimal training data.
  arXiv  Detail & Related papers  (2024-12-20T02:21:36Z)
• Small Agent Can Also Rock! Empowering Small Language Models as Hallucination Detector [114.88975874411142]
  Hallucination detection is a challenging task for large language models (LLMs) We propose an autonomous LLM-based agent framework, called HaluAgent. In HaluAgent, we integrate the LLM, multi-functional toolbox, and design a fine-grained three-stage detection framework.
  arXiv  Detail & Related papers  (2024-06-17T07:30:05Z)
• AvaTaR: Optimizing LLM Agents for Tool Usage via Contrastive Reasoning [93.96463520716759]
  Large language model (LLM) agents have demonstrated impressive capabilities in utilizing external tools and knowledge to boost accuracy and hallucinations. Here, we introduce AvaTaR, a novel and automated framework that optimize an LLM agent to effectively leverage provided tools, improving performance on a given task.
  arXiv  Detail & Related papers  (2024-06-17T04:20:02Z)
• Get my drift? Catching LLM Task Drift with Activation Deltas [55.75645403965326]
  Task drift allows attackers to exfiltrate data or influence the LLM's output for other users.<n>We show that a simple linear classifier can detect drift with near-perfect ROC AUC on an out-of-distribution test set.<n>We observe that this approach generalizes surprisingly well to unseen task domains, such as prompt injections, jailbreaks, and malicious instructions.
  arXiv  Detail & Related papers  (2024-06-02T16:53:21Z)
• Efficient Tool Use with Chain-of-Abstraction Reasoning [63.08202389132155]
  Large language models (LLMs) need to ground their reasoning to real-world knowledge.<n>There remains challenges for fine-tuning LLM agents to invoke tools in multi-step reasoning problems.<n>We propose a new method for LLMs to better leverage tools in multi-step reasoning.
  arXiv  Detail & Related papers  (2024-01-30T21:53:30Z)
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation [96.71370747681078]
  We introduce MLAgentBench, a suite of 13 tasks ranging from improving model performance on CIFAR-10 to recent research problems like BabyLM. For each task, an agent can perform actions like reading/writing files, executing code, and inspecting outputs. We benchmark agents based on Claude v1.0, Claude v2.1, Claude v3 Opus, GPT-4, GPT-4-turbo, Gemini-Pro, and Mixtral and find that a Claude v3 Opus agent is the best in terms of success rate.
  arXiv  Detail & Related papers  (2023-10-05T04:06:12Z)
• SEED: Domain-Specific Data Curation With Large Language Models [22.54280367957015]
  We present SEED, an LLM-as-compiler approach that automatically generates domain-specific data curation solutions via Large Language Models (LLMs) SEED features an that automatically selects from the four LLM-assisted modules and forms a hybrid execution pipeline that best fits the task at hand.
  arXiv  Detail & Related papers  (2023-10-01T17:59:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.