DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learning

• URL: http://arxiv.org/abs/2605.29568v1
• Date: Thu, 28 May 2026 08:17:20 GMT
• Title: DeepTool: Scaling Interleaved Deliberation in Tool-Integrated Reasoning via Process-Supervised Reinforcement Learning
• Abstract summary: We propose a novel framework that scales deliberate thinking within the interleaved process of thinking, action, and observation at each turn.<n>In DeepTool, we first introduce a synthesis pipeline that evolves extended thinking into interleaved trajectories, integrating adversarial perturbations to ensure robustness and self-correction.<n> Secondly, we devise Process-Supervised Reinforcement Learning based on GRPO, which utilizes an Action-Centric Process Reward to reinforce intermediate interleaved thinking.
• Score: 46.01403307350096
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Tool-Integrated Reasoning (TIR) extends LLM capabilities by leveraging external environments. However, existing methods lack the deliberation during sequential tool invocation required for strategic planning and self-correction. While RL mitigates this, conventional approaches for Tool-Integrated Reasoning are hindered by sparse outcome-based rewards, failing to supervise intermediate reasoning steps and tool invocations. To address this, we propose DeepTool, a novel framework that scales deliberate thinking within the interleaved process of thinking, action, and observation at each turn. In DeepTool, we first introduce a synthesis pipeline that evolves extended thinking into interleaved trajectories, integrating adversarial perturbations to ensure robustness and self-correction. Secondly, we devise Process-Supervised Reinforcement Learning based on GRPO, which utilizes an Action-Centric Process Reward to reinforce intermediate interleaved thinking and enforce precise tool invocation at every turn. Extensive experiments demonstrate that DeepTool achieves superior performance, boosting Qwen2.5-7B significantly across six benchmarks (e.g., AIME24: 3.2% -> 40.4% and HMMT25: 0.0% -> 28.6%). Furthermore, the token cost-effectiveness analysis confirms the utility of interleaved thinking, demonstrating DeepTool's optimal balance between performance and token efficiency.

Related papers

• Teaching Thinking Models to Reason with Tools: A Full-Pipeline Recipe for Tool-Integrated Reasoning [59.74608632210439]
  We show how to inject natural tool-use behavior into a strong thinking model without sacrificing its no-tool reasoning ability.<n>Our recipe yields models that achieve state-of-the-art performance in a wide range of benchmarks among open-source models.
  arXiv  Detail & Related papers  (2026-05-07T14:23:21Z)
• ToolOmni: Enabling Open-World Tool Use via Agentic learning with Proactive Retrieval and Grounded Execution [49.30293260493709]
  Large Language Models (LLMs) enhance their problem-solving capability by utilizing external tools.<n>Existing methods relying on static embedding retrieval or parameter memorization struggle to align user intent with tool semantics.<n>We present Tool Omni, a unified agentic framework that enables LLMs for open-world tool use by proactive retrieval and grounded execution.
  arXiv  Detail & Related papers  (2026-04-15T12:26:10Z)
• TInR: Exploring Tool-Internalized Reasoning in Large Language Models [31.2602361688688]
  Tool-Internalized Reasoning (TInR) is a tool-internalized reasoning framework for unified reasoning and tool usage.<n>TInR-U is trained through a three-phase pipeline: 1) tool internalization with a bidirectional knowledge alignment strategy; 2) supervised fine-tuning warm-up using high-quality reasoning annotations; and 3) reinforcement learning with TInR-specific rewards.<n>Experiment results show that TInR-U achieves superior performance in both settings, highlighting its effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-04-12T19:38:19Z)
• Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learning [82.89535601592739]
  We propose a two-stage pipeline that generates reliable base tool-use trajectories through self-evolving synthesis with multi-level validation.<n>These augmentations introduce distractor tools, indirect or ambiguous user queries, and noisy, multi-format, or erroneous tool outputs.<n>This design enables automatic reward computation via reference matching for standard cases and lightweight judge-assisted verification for special behaviors such as error detection.
  arXiv  Detail & Related papers  (2026-04-10T18:38:52Z)
• ZEBRAARENA: A Diagnostic Simulation Environment for Studying Reasoning-Action Coupling in Tool-Augmented LLMs [54.7743875084328]
  We introduce ZebraArena, a diagnostic environment for studying reasoning-action coupling in tool-augmented large language models.<n>Each task in ZebraArena requires a set of critical information which is available only through targeted tool use.<n>We show that ZebraArena requires a combination of in-depth reasoning and accurate external tool calling, which remains a challenge.
  arXiv  Detail & Related papers  (2026-03-19T08:33:54Z)
• Discovery and Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees [16.028598757687558]
  We introduce DART (Discovery And Reinforcement of Tool-Integrated Reasoning Chains via Rollout Trees), a reinforcement learning framework.<n>DART operates by constructing dynamic rollout trees during training to discover valid tool-use opportunities.<n>Tree-based process advantage estimation identifies and credits specific sub-trajectories where tool invocation positively contributes to the solution.
  arXiv  Detail & Related papers  (2026-01-13T07:06:21Z)
• Acting Less is Reasoning More! Teaching Model to Act Efficiently [87.28134636548705]
  Tool-integrated reasoning augments large language models with the ability to invoke external tools to solve tasks.<n>Current approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use.<n>We propose a framework that encourages models to produce accurate answers with minimal tool calls.<n>Our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.
  arXiv  Detail & Related papers  (2025-04-21T05:40:05Z)
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs [27.07998056454784]
  ReTool enhances long-form reasoning with tool-integrated learning.<n>Model achieves 67% accuracy with 400 training steps.<n>Remarkably, ReTool-32B attains 72.5% accuracy in extended settings.
  arXiv  Detail & Related papers  (2025-04-15T18:10:22Z)
• ToRL: Scaling Tool-Integrated RL [25.477841726836836]
  ToRL is a framework for training large language models to autonomously use computational tools.<n>ToRL allows models to explore and discover optimal strategies for tool use.<n>Experiments with Qwen2.5-Math models show significant improvements.
  arXiv  Detail & Related papers  (2025-03-30T10:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.