When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents

• URL: http://arxiv.org/abs/2605.11928v1
• Date: Tue, 12 May 2026 10:40:28 GMT
• Title: When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents
• Abstract summary: We study failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP)<n>We propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories.<n>On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate accuracy comparable to open-source 14B function-calling baselines.
• Score: 32.54888729689753
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tool-use language agents are evaluated on benchmarks that assume clean inputs, unambiguous tool registries, and reliable APIs. Real deployments violate all these assumptions: user typos propagate into hallucinated tool names, a misconfigured request timeout can stall an agent indefinitely, and duplicate tool names across servers can freeze an SDK. We study these failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP), where deployment noise enters through the observation, action space, reward-relevant metadata, or transition dynamics. We introduce RobustBench-TC, a benchmark with 22 perturbation types organized by these four POMDP components, each grounded in a verified GitHub issue or documented tool-calling failure. Across 21 models from 1.5B to 32B parameters (including the closed-source o4-mini), the robustness profile is sharply uneven: observation perturbations reduce accuracy by less than 5%, while reward-relevant and transition perturbations reduce accuracy by roughly 40% and 30%, respectively; scale alone does not close these gaps. We then propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories spanning the three statically encodable POMDP components. On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate perturbed accuracy comparable to open-source 14B function-calling baselines while substantially narrowing the gap to o4-mini. It closes approximately 27% of the Transition gap despite never seeing transition perturbations in training, suggesting that RL on adversarial static tool-use inputs induces a more persistent retry policy that transfers to unseen runtime failures. The dataset, code and benchmark leaderboard are publicly available.

Related papers

• Joint Optimization of Tool Creation and Use for Large Language Model Agents [69.07066297088187]
  SMITH (-grounded Multi-task Iterative Tool Honing) trains tool creation and tool use inside a single policy.<n>A 4B Qwen3 trained with SMITH on 13 procedural reasoning tasks with exact verifiers reaches 79.8 macro-average accuracy on held-out tasks.<n>It also reaches 40.4 on TabMWP-Hard and 42.6 on out-of-domain GQA (+7.6 over the best same-backbone inference-time baseline)
  arXiv  Detail & Related papers  (2026-08-25T13:59:34Z)
• ToolRobustBench: Stage-Wise Perturbation Evaluation and Failure Diagnosis for Tool-Calling Agents [16.995082216096787]
  Large language models rely on tool calling as a fundamental agent capability.<n>Clean end-to-end (E2E) success cannot identify where a tool-use failure originates or how it propagates through a call.<n>We introduce ToolRobustBench, a stage-wise diagnostic benchmark for tool-calling agents.
  arXiv  Detail & Related papers  (2026-08-23T15:22:49Z)
• AgentDebugX: An Open-Source Toolkit for Failure Observability, Attribution, and Recovery in LLM Agents [83.37969790806069]
  LLM agent failures are difficult to debug because the step where an error surfaces is often not the one that caused it.<n>We present Agent DebuggerX, an open-source framework that organizes debug as a closed loop of Detect, Attribute, Recover, and Rerun.<n>At its core, Deep Debugger performs multi-turn root-cause diagnosis through global trajectory understanding.
  arXiv  Detail & Related papers  (2026-07-21T06:21:13Z)
• HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents [52.33697136670754]
  We introduce textbfHyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution.<n>A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally.<n>On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average
  arXiv  Detail & Related papers  (2026-06-11T17:56:36Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Agent Explorative Policy Optimization for Multimodal Agentic Reasoning [97.64835302176056]
  Agentic reasoning interleaves two behaviors with a structural asymmetry: thinking and tool use.<n>Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training.<n>We propose AXPO (Agent eXplorative Policy Optimization) for each all-wrong tool-using subgroup.
  arXiv  Detail & Related papers  (2026-05-27T17:36:39Z)
• TRACER: Verifiable Generative Provenance for Multimodal Tool-Using Agents [19.156453695628013]
  We introduce TRACER, a framework for verifiable generative provenance in multimodal tool-using agents.<n>TraCER generates each answer sentence together with a structured provenance record that identifies the supporting tool turn, evidence unit, and semantic support relation.<n>With Qwen3-VL-8B, TRACER reaches 78.23% answer accuracy and 95.72% summary accuracy, outperforming the strongest closed-source tool-augmented baseline by 23.80 percentage points.
  arXiv  Detail & Related papers  (2026-05-11T03:32:55Z)
• Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning [82.89535601592739]
  We propose a two-stage pipeline that generates reliable base tool-use trajectories through self-evolving synthesis with multi-level validation.<n>These augmentations introduce distractor tools, indirect or ambiguous user queries, and noisy, multi-format, or erroneous tool outputs.<n>This design enables automatic reward computation via reference matching for standard cases and lightweight judge-assisted verification for special behaviors such as error detection.
  arXiv  Detail & Related papers  (2026-04-10T18:38:52Z)
• Dynamic analysis enhances issue resolution [53.50448142467294]
  DAIRA (Dynamic Analysis-enhanced Issue Resolution Agent) is an automated repair framework that embeds dynamic analysis into the agent's reasoning cycle.<n> Driven by a Test Tracing-Driven methodology, DAIRA utilizes lightweight monitors to extract critical runtime data.<n>When powered by Gemini 3 Flash Preview, DAIRA establishes a new state-of-the-art (SOTA) performance, achieving a 79.4% resolution rate on the SWE-bench Verified dataset.
  arXiv  Detail & Related papers  (2026-03-23T14:48:54Z)
• AgentProcessBench: Diagnosing Step-Level Process Quality in Tool-Using Agents [50.481033105867205]
  We introduce AgentProcessBench, the first benchmark dedicated to evaluating step-level effectiveness in realistic, tool-augmented trajectories.<n>The benchmark comprises 1,000 diverse trajectories and 8,509 human-labeled step annotations with 89.1% inter-annotator agreement.<n>It features a ternary labeling scheme to capture exploration and an error propagation rule to reduce labeling ambiguity.
  arXiv  Detail & Related papers  (2026-03-15T16:13:58Z)
• RIVA: Leveraging LLM Agents for Reliable Configuration Drift Detection [3.494935876363005]
  Existing agentic systems implicitly assume that the tools they invoke always return correct outputs, making them vulnerable to erroneous tool responses.<n>We introduce RIVA, a novel multi-agent system that performs robust IaC verification even when tools produce incorrect or misleading outputs.<n>Our results show that cross-validation of diverse tool calls enables more reliable autonomous infrastructure verification in production cloud environments.
  arXiv  Detail & Related papers  (2026-03-02T19:28:27Z)
• Robust Tool Use via Fission-GRPO: Learning to Recover from Execution Errors [41.78467154106763]
  We propose Fission-GRPO, a framework that converts execution errors into corrective supervision within the RL training loop.<n>Our core mechanism fissions each failed trajectory into a new training instance by augmenting it with diagnostic feedback from a finetuned Error Simulator.<n>On the BFCL v4 Multi-Turn, Fission-GRPO improves the error recovery rate of Qwen3-8B by 5.7% absolute, crucially, yielding a 4% overall accuracy gain.
  arXiv  Detail & Related papers  (2026-01-22T03:57:35Z)
• ET-Agent: Incentivizing Effective Tool-Integrated Reasoning Agent via Behavior Calibration [68.89572566071575]
  ETAgent is a training framework for calibrating agent's tool-use behavior.<n>It is designed to progressively calibrate erroneous behavioral patterns to optimal behaviors.
  arXiv  Detail & Related papers  (2026-01-11T11:05:26Z)
• Adaptive Monitoring and Real-World Evaluation of Agentic AI Systems [3.215065407261898]
  Multi-agent systems that combine large language models with external tools are rapidly transitioning from research laboratories into high-stakes domains.<n>This "Advanced" sequel fills that gap by providing an algorithmic instantiation or empirical evidence.<n>AMDM cuts anomaly-detection latency from 12.3 s to 5.6 s on simulated goal drift and reduces false-positive rates from 4.5% to 0.9%.
  arXiv  Detail & Related papers  (2025-08-28T15:52:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.