SynthTools: A Framework for Scaling Synthetic Tools for Agent Development

• URL: http://arxiv.org/abs/2511.09572v1
• Date: Fri, 14 Nov 2025 01:00:22 GMT
• Title: SynthTools: A Framework for Scaling Synthetic Tools for Agent Development
• Abstract summary: We introduce SynthTools, a flexible and scalable framework for generating synthetic tool ecosystems.<n>Our framework consists of three core components: Tool Generation for automatic and scalable creation of diverse tools, Tool Simulation to emulate realistic tool behaviors, and Tool Audit to ensure correctness and consistency of tool simulation.<n>By enabling scalable, diverse, and reliable tool ecosystems, SynthTools provides a practical path toward large-scale training and stable evaluation of tool-use agents.
• Score: 5.411691071249542
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: AI agents increasingly rely on external tools to solve complex, long-horizon tasks. Advancing such agents requires reproducible evaluation and large-scale training in controllable, diverse, and realistic tool-use environments. However, real-world APIs are limited in availability, domain coverage, and stability, often requiring access keys and imposing rate limits, which render them impractical for stable evaluation or scalable training. To address these challenges, we introduce SynthTools, a flexible and scalable framework for generating synthetic tool ecosystems. Our framework consists of three core components: Tool Generation for automatic and scalable creation of diverse tools, Tool Simulation to emulate realistic tool behaviors, and Tool Audit to ensure correctness and consistency of tool simulation. To illustrate its scalability, we show that SynthTools can readily produce toolsets that span twice as many domains and twice as many tools per domain as prior work. Furthermore, the tool simulation and tool audit components demonstrate strong reliability, achieving $94\%$ and $99\%$ accuracy respectively. Finally, we construct downstream tasks from the generated tools that even state-of-the-art models struggle to complete. By enabling scalable, diverse, and reliable tool ecosystems, SynthTools provides a practical path toward large-scale training and stable evaluation of tool-use agents. Our code is available at https://github.com/namkoong-lab/SynthTools.

Related papers

• SciToolAgent-Evo: An Ontology-Aware Self-Evolving Agent for Open-World Scientific Tool Acquisition [53.36281538053445]
  We propose SciToolAgent-Evo, an ontology-aware self-evolving agent for open-world scientific tool acquisition.<n> Driven by an evolving memory of skills, experiences, and an ontologized tool graph, it distills generalizable knowledge from contrastive trajectories during accumulation.<n>Once a novel tool is acquired, its scientific knowledge is completed online for seamless integration into the known graph.
  arXiv  Detail & Related papers  (2026-07-30T08:47:11Z)
• SING: Synthetic Intention Graph for Scalable Active Tool Discovery in LLM Agents [64.59100414726556]
  Large language model (LLM) agents increasingly rely on harnesses that manage context, tools, and multi-turn execution.<n>Retrieval-augmented tool selection offers a natural alternative, but existing one-shot retrieval methods fail to align isolated tool descriptions with the agent's true task intention.<n>We propose SING, an intention-aware active tool discovery framework that builds an intention-tool graph linking user intentions, tool capabilities, and tool collaboration patterns.
  arXiv  Detail & Related papers  (2026-06-15T11:37:37Z)
• Open, Reliable, and Collective: A Community-Driven Framework for Tool-Using AI Agents [22.64138018985385]
  We argue that failures stem from both tool-use accuracy (how well an agent invokes a tool) and intrinsic tool accuracy (the tool's own correctness)<n>We introduce OpenTools, a community-driven toolbox that standardizes tool schemas.<n>OpenTools includes the core framework, an initial tool set, evaluation pipelines, and a contribution protocol.
  arXiv  Detail & Related papers  (2026-03-31T18:42:36Z)
• FinToolSyn: A forward synthesis Framework for Financial Tool-Use Dialogue Data with Dynamic Tool Retrieval [59.67674731525128]
  FinToolSyn is a forward synthesis framework designed to generate high-quality financial dialogues.<n>Our pipeline constructs a repository of 43,066 tools and synthesizes over 148k dialogue instances.<n>Extensive experiments demonstrate that models trained on FinToolSyn achieve a 21.06% improvement.
  arXiv  Detail & Related papers  (2026-03-25T08:00:26Z)
• The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestration [58.61149924275458]
  Tool use enables large language models to access external information, invoke software systems, and act in digital environments beyond what can be solved from model parameters alone.<n>As agent systems evolve, the central problem has shifted from isolated invocation to multi-tool orchestration over long trajectories with intermediate state, execution feedback, changing environments, and practical constraints such as safety, cost, and verifiability.<n>We organize the literature around six core dimensions: inference-time planning and execution, training and trajectory construction, safety and control, efficiency under resource constraints, capability completeness in open environments, and benchmark design and evaluation.
  arXiv  Detail & Related papers  (2026-03-24T07:05:05Z)
• Tool-Genesis: A Task-Driven Tool Creation Benchmark for Self-Evolving Language Agent [45.450766613995135]
  Tool-Genesis is a diagnostic benchmark designed to quantify agent capabilities across multiple dimensions.<n>We find that even state-of-the-art models struggle to produce precise tool interfaces or executable logic in a one-shot setting.
  arXiv  Detail & Related papers  (2026-03-05T17:44:29Z)
• ToolMATH: A Math Tool Benchmark for Realistic Long-Horizon Multi-Tool Reasoning [11.99927786717109]
  ToolMATH turns math problems into a controlled, correctness-checkable benchmark with tool sets.<n>ToolMATH provides actionable diagnostic evidence of failure modes in tool-augmented agents.
  arXiv  Detail & Related papers  (2026-02-24T09:23:12Z)
• Learning to Rewrite Tool Descriptions for Reliable LLM-Agent Tool Use [21.666294374943178]
  We propose a curriculum learning framework that transfers supervision from trace-rich settings to trace-free deployment.<n> Experiments show consistent gains on unseen tools, strong cross-domain generalization, and robustness as the number of candidate tools scales to over 100.
  arXiv  Detail & Related papers  (2026-02-23T23:50:24Z)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning [62.499592503950026]
  Large language model (LLM) have empowered autonomous agents to perform complex tasks that require multi-turn interactions with tools and environments.<n>We propose Agent World Model (AWM), a fully synthetic environment generation pipeline.<n>We scale to 1,000 environments covering everyday scenarios, in which agents can interact with rich toolsets.
  arXiv  Detail & Related papers  (2026-02-10T18:55:41Z)
• ToolTok: Tool Tokenization for Efficient and Generalizable GUI Agents [16.06309106596998]
  ToolTok is a novel paradigm of multi-step pathfinding for GUI agents.<n>We devise tools aligned with human interaction habits and represent each tool using learnable token embeddings.<n>We construct an easy-to-hard curriculum consisting of three tasks: token definition question-answering, pure text-guided tool selection, and simplified visual pathfinding.
  arXiv  Detail & Related papers  (2026-01-30T08:38:05Z)
• AdaReasoner: Dynamic Tool Orchestration for Iterative Visual Reasoning [66.24374176797075]
  We introduce textbfAdaReasoner, a family of multimodal models that learn tool use as a general reasoning skill rather than as tool-specific or explicitly supervised behavior.<n>AdaReasoner is enabled by (i) a scalable data curation pipeline exposing models to long-horizon, multi-step tool interactions; (ii) Tool-GRPO, a reinforcement learning algorithm that prioritizes tool selection and sequencing based on end-task success; and (iii) an adaptive learning mechanism that dynamically regulates tool usage.
  arXiv  Detail & Related papers  (2026-01-26T16:04:43Z)
• ToolLibGen: Scalable Automatic Tool Creation and Aggregation for LLM Reasoning [80.10274552177096]
  Large Language Models (LLMs) equipped with external tools have demonstrated enhanced performance on complex reasoning tasks.<n>The widespread adoption of this tool-augmented reasoning is hindered by the scarcity of domain-specific tools.<n>We propose a systematic approach to automatically an unstructured collection of tools into a structured tool library.
  arXiv  Detail & Related papers  (2025-10-09T04:11:16Z)
• Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use [50.02614257515131]
  Large language models (LLMs) have demonstrated strong capabilities in language understanding and reasoning.<n>We propose Tool-R1, a reinforcement learning framework that enables LLMs to perform general, compositional, and multi-step tool use.
  arXiv  Detail & Related papers  (2025-09-16T09:22:21Z)
• VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use [78.29315418819074]
  We introduce VerlTool, a unified and modular framework that addresses limitations through systematic design principles.<n>Our framework formalizes ARLT as multi-turn trajectories with multi-modal observation tokens (text/image/video), extending beyond single-turn RLVR paradigms.<n>The modular plugin architecture enables rapid tool integration requiring only lightweight Python definitions.
  arXiv  Detail & Related papers  (2025-09-01T01:45:18Z)
• Feedback-Driven Tool-Use Improvements in Large Language Models via Automated Build Environments [70.42705564227548]
  We propose an automated environment construction pipeline for large language models (LLMs)<n>This enables the creation of high-quality training environments that provide detailed and measurable feedback without relying on external tools.<n>We also introduce a verifiable reward mechanism that evaluates both the precision of tool use and the completeness of task execution.
  arXiv  Detail & Related papers  (2025-08-12T09:45:19Z)
• ToolGen: Unified Tool Retrieval and Calling via Generation [34.34787641393914]
  We introduce ToolGen, a paradigm shift that integrates tool knowledge directly into the large language models' parameters.<n>We show that ToolGen achieves superior results in both tool retrieval and autonomous task completion.<n>ToolGen paves the way for more versatile, efficient, and autonomous AI systems.
  arXiv  Detail & Related papers  (2024-10-04T13:52:32Z)
• MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation [25.360660222418183]
  We present MetaTool, a novel tool learning methodology designed to generalize across any reusable toolset. By incorporating meta-task data into task-oriented training, our method significantly enhances the performance of open-source Large Language Models.
  arXiv  Detail & Related papers  (2024-07-15T10:15:41Z)
• Planning, Creation, Usage: Benchmarking LLMs for Comprehensive Tool Utilization in Real-World Complex Scenarios [93.68764280953624]
  UltraTool is a novel benchmark designed to improve and evaluate Large Language Models' ability in tool utilization. It emphasizes real-world complexities, demanding accurate, multi-step planning for effective problem-solving. A key feature of UltraTool is its independent evaluation of planning with natural language, which happens before tool usage.
  arXiv  Detail & Related papers  (2024-01-30T16:52:56Z)
• Making Language Models Better Tool Learners with Execution Feedback [36.30542737293863]
  Tools serve as pivotal interfaces that enable humans to understand and reshape the environment. Existing tool learning methodologies induce large language models to utilize tools indiscriminately. We propose Tool leaRning wIth exeCution fEedback (TRICE), a two-stage end-to-end framework that enables the model to continually learn through feedback derived from tool execution.
  arXiv  Detail & Related papers  (2023-05-22T14:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.