Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning

• URL: http://arxiv.org/abs/2608.28447v1
• Date: Fri, 28 Aug 2026 15:35:03 GMT
• Title: Learning to Use Tools: Reinforcement Learning for Tool-Integrated Mathematical Reasoning
• Abstract summary: We study calculator tool calling for improving mathematical reasoning on the Countdown task.<n>We apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO.<n>Results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k.
• Score: 8.739632908951696
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Current large language models (LLMs) increasingly benefit from external tool integration, especially for tasks requiring reliable computation and verification. Motivated by this, we study calculator tool calling for improving mathematical reasoning on the Countdown task. We first analyze reasoning failures and find that calculation errors account for a substantial portion of incorrect responses. We then construct supervised fine-tuning datasets to teach the model useful tool-use patterns and how to interpret returned outputs. Building on this tool-formatted policy, we apply several on-policy reinforcement learning methods, including RLOO, RLOO++, GRPO, and DAPO, using automatically verifiable final-answer rewards. To enable a more reliable evaluation, we construct a fresh 1,024-problem held-out Countdown benchmark with no exact overlap with the training data. Our results show that calculator tool integration consistently improves both SFT and RL baselines, yielding roughly 10 percentage-point gains across pass@k. Among the RL methods, Tool-DAPO achieves the strongest performance, improving pass@1 from 35.8% for Tool-SFT to 66.0%. Further analysis shows that RL encourages more effective tool use even when only final-answer rewards are provided. These findings suggest that tool integration reduces arithmetic and verification errors, while RL increases the probability of correct reasoning traces.

Related papers

• Learning When Not to Act: Mitigating Tool Abuse in Agentic Reinforcement Learning [26.34952204312613]
  Agentic reinforcement learning can induce tool abuse, where models overuse external tools even for solvable queries by internal reasoning.<n>We propose EAPO, an Efficient Agentic Policy Optimization framework that learns selective tool use.<n>Compared with GRPO, EAPO improves average performance by 10.45%, 7.27%, and 9.69%, while reducing average tool calls by 18.33%, 18.33%, and 24.59%, respectively.
  arXiv  Detail & Related papers  (2026-06-01T11:58:55Z)
• When to Trust Tools? Adaptive Tool Trust Calibration For Tool-Integrated Math Reasoning [58.75883713573783]
  Tool-Integrated Reasoning has emerged as a promising paradigm that incorporates tool call and execution within the reasoning trajectory.<n>We find that when the reasoning of the model conflicts with the tool results, the model tends to believe in its own reasoning.<n>We introduce Adaptive Tool Trust (ATTC), a framework that guides the model to adaptively choose to trust or ignore the tool results.
  arXiv  Detail & Related papers  (2026-04-09T14:14:37Z)
• Tool-R1: Sample-Efficient Reinforcement Learning for Agentic Tool Use [50.02614257515131]
  Large language models (LLMs) have demonstrated strong capabilities in language understanding and reasoning.<n>We propose Tool-R1, a reinforcement learning framework that enables LLMs to perform general, compositional, and multi-step tool use.
  arXiv  Detail & Related papers  (2025-09-16T09:22:21Z)
• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training [121.5858973157225]
  We investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains.<n>We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains.<n>Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks.
  arXiv  Detail & Related papers  (2025-07-16T17:59:24Z)
• Advancing Tool-Augmented Large Language Models via Meta-Verification and Reflection Learning [63.2198957755528]
  We propose Tool-MVR, a novel Tool-Augmented LLM that achieves comprehensive System 2 reasoning through two key innovations.<n>Specifically, we first introduce Multi-Agent Meta-Verification (MAMV), a systematic pipeline that rigorously validates APIs, queries, and reasoning trajectories.<n>Second, we propose Exploration-based Reflection Learning (EXPLORE), which enhances tool reflection capabilities by leveraging tool feedback.
  arXiv  Detail & Related papers  (2025-06-05T04:35:49Z)
• Agent RL Scaling Law: Agent RL with Spontaneous Code Execution for Mathematical Problem Solving [26.413753656936688]
  Large Language Models (LLMs) often struggle with mathematical reasoning tasks requiring precise, verifiable computation.<n>While Reinforcement Learning (RL) from outcome-based rewards enhances text-based reasoning, understanding how agents autonomously learn to leverage external tools like code execution remains crucial.
  arXiv  Detail & Related papers  (2025-05-12T17:23:34Z)
• Acting Less is Reasoning More! Teaching Model to Act Efficiently [87.28134636548705]
  Tool-integrated reasoning augments large language models with the ability to invoke external tools to solve tasks.<n>Current approaches typically optimize only for final correctness without considering the efficiency or necessity of external tool use.<n>We propose a framework that encourages models to produce accurate answers with minimal tool calls.<n>Our approach reduces tool calls by up to 68.3% and improves tool productivity by up to 215.4%, while maintaining comparable answer accuracy.
  arXiv  Detail & Related papers  (2025-04-21T05:40:05Z)
• ToolRL: Reward is All Tool Learning Needs [54.16305891389931]
  Large Language Models (LLMs) often undergo supervised fine-tuning (SFT) to acquire tool use capabilities.<n>Recent advancements in reinforcement learning (RL) have demonstrated promising reasoning and generalization abilities.<n>We present the first comprehensive study on reward design for tool selection and application tasks within the RL paradigm.
  arXiv  Detail & Related papers  (2025-04-16T21:45:32Z)
• ReTool: Reinforcement Learning for Strategic Tool Use in LLMs [27.07998056454784]
  ReTool enhances long-form reasoning with tool-integrated learning.<n>Model achieves 67% accuracy with 400 training steps.<n>Remarkably, ReTool-32B attains 72.5% accuracy in extended settings.
  arXiv  Detail & Related papers  (2025-04-15T18:10:22Z)
• ToRL: Scaling Tool-Integrated RL [25.477841726836836]
  ToRL is a framework for training large language models to autonomously use computational tools.<n>ToRL allows models to explore and discover optimal strategies for tool use.<n>Experiments with Qwen2.5-Math models show significant improvements.
  arXiv  Detail & Related papers  (2025-03-30T10:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.