The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior

• URL: http://arxiv.org/abs/2608.11386v1
• Date: Tue, 11 Aug 2026 19:50:56 GMT
• Title: The Devil Is in the Interface: Evaluating How Tool Architecture Shapes Coding Agent Behavior
• Abstract summary: We study tool architecture in coding agents through controlled experiments on repository-level issue fixing.<n>Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior.
• Score: 27.33402415317721
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As large language models continue to improve, agentic systems are becoming increasingly important, and tools are a key design dimension because they determine how agents access information and take action in their environments. Prior work on agent tooling has primarily focused on expanding what agents can do, but has paid less systematic attention to how those capabilities are organized and exposed to the model. We refer to this latter design dimension as tool architecture. We study tool architecture in coding agents through controlled experiments on repository-level issue fixing, comparing six tool architectures that hold the underlying information and actions similar while varying how they are organized and exposed to the model, across three actors and a total of 11,700 trajectories. Our experiments show that, even when tools provide similar capabilities, tool architecture changes agent behavior: Compared to a basic architecture where the agent has only the bash tool, more structured low-level interfaces improve consistency across repeated attempts by up to 4.7 $\times$; natural-language search broadens repository exploration and increases access to relevant files by more than 11%; and Python CodeAct-style interfaces achieve similar task performance with 41.6% fewer steps and 56.3% lower token usage. By contrast, lightweight text-based cognitive-scaffolding tools, such as tools that let the agent record intermediate reasoning, have limited effect on actor behavior.

Related papers

• ToolAtlas: Learning Once, Reusing Everywhere with Tool-Side Memory [49.08868447320197]
  Large language model (LLM) agents increasingly rely on external tools served by shared providers.<n>Existing approaches improve tool use on the agent side through parameter updates, prompt refinement, or agent-side memory.<n>We introduce ToolAtlas, a graph-based framework that builds a persistent provider-side tool memory of tool capabilities.
  arXiv  Detail & Related papers  (2026-07-13T06:01:41Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• Exploring Interaction Paradigms for LLM Agents in Scientific Visualization [7.22431217973039]
  This paper examines how different types of large language model (LLM) agents perform on scientific visualization (SciVis) tasks.<n>We compare three primary interaction paradigms, including domain-specific agents with structured tool use, computer-use agents, and general-purpose coding agents.
  arXiv  Detail & Related papers  (2026-04-30T15:22:28Z)
• Semantic Tool Discovery for Large Language Models: A Vector-Based Approach to MCP Tool Selection [1.2633421050364024]
  Large Language Models (LLMs) with tool-calling capabilities have demonstrated remarkable potential in executing complex tasks through external tool integration.<n>The Model Context Protocol (MCP) has emerged as a standardized framework for connecting LLMs to diverse toolsets, with individual MCP servers potentially exposing dozens to hundreds of tools.<n>We present a semantic tool discovery architecture that addresses these challenges through vector-based retrieval.
  arXiv  Detail & Related papers  (2026-03-19T18:10:35Z)
• DeepAgent: A General Reasoning Agent with Scalable Toolsets [111.6384541877723]
  DeepAgent is an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution.<n>To address the challenges of long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories.<n>We develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens.
  arXiv  Detail & Related papers  (2025-10-24T16:24:01Z)
• TheMCPCompany: Creating General-purpose Agents with Task-specific Tools [12.249551019598442]
  We introduce TheMCPCompany, a benchmark for evaluating tool-calling agents on tasks that involve interacting with various real-world services.<n>We also provide manually annotated ground-truth tools for each task.<n>Overall, our work shows that the most advanced reasoning models are effective at discovering tools in simpler environments, but seriously struggle with navigating complex enterprise environments.
  arXiv  Detail & Related papers  (2025-10-22T06:42:01Z)
• ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM Reasoning [80.10274552177096]
  Large Language Models (LLMs) equipped with external tools have demonstrated enhanced performance on complex reasoning tasks.<n>The widespread adoption of this tool-augmented reasoning is hindered by the scarcity of domain-specific tools.<n>We propose a systematic approach to automatically an unstructured collection of tools into a structured tool library.
  arXiv  Detail & Related papers  (2025-10-09T04:11:16Z)
• NaviAgent: Bilevel Planning on Tool Navigation Graph for Large-Scale Orchestration [13.925896302382043]
  Large language models (LLMs) have recently demonstrated the ability to act as function call agents by invoking external tools.<n>We propose NaviAgent, which decouples task planning from tool execution through graph-based modeling of the tool ecosystem.<n> Experiments show that NaviAgent achieves the best task success rates across models and tasks, and integrating TWMN further boosts performance by up to 17 points on complex tasks.
  arXiv  Detail & Related papers  (2025-06-24T10:39:07Z)
• SMART: Self-Aware Agent for Tool Overuse Mitigation [58.748554080273585]
  Current Large Language Model (LLM) agents demonstrate strong reasoning and tool use capabilities, but often lack self-awareness.<n>This imbalance leads to Tool Overuse, where models unnecessarily rely on external tools for tasks with parametric knowledge.<n>We introduce SMART (Strategic Model-Aware Reasoning with Tools), a paradigm that enhances an agent's self-awareness to optimize task handling and reduce tool overuse.
  arXiv  Detail & Related papers  (2025-02-17T04:50:37Z)
• AgentStudio: A Toolkit for Building General Virtual Agents [57.02375267926862]
  General virtual agents need to handle multimodal observations, master complex action spaces, and self-improve in dynamic, open-domain environments.<n>AgentStudio provides a lightweight, interactive environment with highly generic observation and action spaces.<n>It integrates tools for creating online benchmark tasks, annotating GUI elements, and labeling actions in videos.<n>Based on our environment and tools, we curate an online task suite that benchmarks both GUI interactions and function calling with efficient auto-evaluation.
  arXiv  Detail & Related papers  (2024-03-26T17:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.