Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration

• URL: http://arxiv.org/abs/2604.02869v1
• Date: Fri, 03 Apr 2026 08:36:03 GMT
• Title: Multi-Turn Reinforcement Learning for Tool-Calling Agents with Iterative Reward Calibration
• Abstract summary: Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging.<n>We present the first application of MT-GRPO and GTPO for training a tool-calling agent on realistic customer service tasks.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Training tool-calling agents with reinforcement learning on multi-turn tasks remains challenging due to sparse outcome rewards and difficult credit assignment across conversation turns. We present the first application of MT-GRPO (Multi-Turn Group Relative Policy Optimization) combined with GTPO (Generalized Token-level Policy Optimization) for training a tool-calling agent on realistic customer service tasks with an LLM-based user simulator. Through systematic analysis of training rollouts, we discover that naively designed dense per-turn rewards degrade performance by up to 14 percentage points due to misalignment between reward discriminativeness and advantage direction. We introduce Iterative Reward Calibration, a methodology for designing per-turn rewards using empirical discriminative analysis of rollout data, and show that our GTPO hybrid advantage formulation eliminates the advantage misalignment problem. Applied to the Tau-Bench airline benchmark, our approach improves Qwen3.5-4B from 63.8 percent to 66.7 percent (+2.9pp) and Qwen3-30B-A3B from 58.0 percent to 69.5 percent (+11.5pp) -- with the trained 4B model exceeding GPT-4.1 (49.4 percent) and GPT-4o (42.8 percent) despite being 50 times smaller, and the 30.5B MoE model approaching Claude Sonnet 4.5 (70.0 percent). To our knowledge, these are the first published RL training results on Tau-Bench. We release our code, reward calibration analysis, and training recipes.

Related papers

• Meta-Learned Reward Shaping for Reinforcement Learning from Human Feedback [0.0]
  We introduce MeRLa (Meta-Learned Reward Shaping), a framework that meta-learns a task-aware shaping function.<n>The learned shaping produces a composite reward that preserves policy optimality.<n>MeRLa retains its benefits when combined with process-based and principled-based enhanced rewards.
  arXiv  Detail & Related papers  (2026-07-28T02:09:26Z)
• Less Data, Better Alignment: Data-Centric Multi-Evaluator Agreement for Preference Optimization [20.147316178118732]
  DMAPO generates candidate responses from the target policy, evaluates helpfulness, factuality, and conciseness with rubric-specialized evaluators.<n>This procedure accepts 1,871 of 54,236 Mistral-7B candidates (3.45%)<n>KTO trained on this set reaches 7.50 on MT-Bench, 95.5% length-controlled win rate against a text-davinci-003 reference, and 57.3% IFEval prompt accuracy.
  arXiv  Detail & Related papers  (2026-07-27T23:05:12Z)
• Mach-Mind-4-Flash Technical Report [0.0]
  Mach-Mind-4-Flash is a Mixture-of-Experts (MoE) agentic model with 3B activated parameters.<n>Post-training optimization alone without scaling pre-training compute, the model achieves performance on par with or surpassing that of 100B- parameter-class models.
  arXiv  Detail & Related papers  (2026-07-10T12:57:08Z)
• expo: Exploration-prioritized policy optimization via adaptive kl regulation and gaussian curriculum sampling [11.537163059885687]
  We propose Exploration-Prioritized Policy Optimization (EXPO) with two lightweight plug-in modules.<n>We conduct experiments on DeepSeek-R1-Distill-Qwen-1.5B and Qwen3-8B-Base over six mathematical reasoning benchmarks.<n>It obtains an absolute gain of 13.34 on AIME 2025 pass@32, rising from 63.33 percent to 76.67 percent, and achieves an average pass@32 improvement of 2.66 on the 8B model.
  arXiv  Detail & Related papers  (2026-05-11T03:19:04Z)
• AgentHER: Hindsight Experience Replay for LLM Agent Trajectory Relabeling [12.42698406379544]
  AgentHER recovers lost training signal by adapting the Hindsight Experience principle to natural-language agent trajectories.<n>It converts discarded failures into high-quality SFT, DPO, and ShareGPT training data.<n>On WebArena Replay (Zhou et al., 2024) and ToolBench (Qin et al., 2024), AgentHER improves over success-only SFT by +7.1-11.7 pp across four model families.
  arXiv  Detail & Related papers  (2026-03-22T18:36:58Z)
• EvasionBench: Detecting Evasive Answers in Financial Q&A via Multi-Model Consensus and LLM-as-Judge [8.50639201265868]
  We introduce EvasionBench, comprising 30,000 training samples and 1,000 human-annotated test samples.<n>We mine boundary cases where two strong annotators conflict, using a judge to resolve labels.<n>Our trained model Eva-4B (4B parameters) achieves 81.3 percent accuracy, outperforming its base by 25 percentage points.
  arXiv  Detail & Related papers  (2026-01-14T04:26:43Z)
• SPARK: Stepwise Process-Aware Rewards for Reference-Free Reinforcement Learning [39.1720897614261]
  Process reward models (PRMs) that provide dense, step-level feedback have shown promise for reinforcement learning.<n>We propose SPARK: a three-stage framework where in the first stage a generator model produces diverse solutions and a verifier model evaluates them.<n>We show that aggregating multiple independent verifications at the step level produces training data for process reward models that surpass ground-truth outcome supervision.
  arXiv  Detail & Related papers  (2025-12-02T21:30:47Z)
• MapReduce LoRA: Advancing the Pareto Front in Multi-Preference Optimization for Generative Models [52.32146943039743]
  Reinforcement learning from human feedback has advanced alignment of generative models to human aesthetic and perceptual preferences.<n>We introduce two complementary methods: MapReduce LoRA and Reward-aware Token Embedding (RaTE)<n>Our framework sets a new state-of-the-art multi-preference alignment recipe across modalities.
  arXiv  Detail & Related papers  (2025-11-25T18:49:21Z)
• Squeeze the Soaked Sponge: Efficient Off-policy Reinforcement Finetuning for Large Language Model [56.92219181993453]
  We propose Reincarnating Mix-policy Proximal Policy Gradient (ReMix) to enable on-policyRFT methods like PPO and GRPO to leverage off-policy data.<n>ReMix consists of three major components: (1) Mix-policy proximal policy gradient with an increased Update-To-Data (UTD) ratio for efficient training; (2) KL-Convex policy constraint to balance the trade-off between stability and flexibility; (3) Policy reincarnation to achieve a seamless transition from efficient early-stage learning to steady improvement.
  arXiv  Detail & Related papers  (2025-07-09T14:29:45Z)
• Pre-Trained Policy Discriminators are General Reward Models [81.3974586561645]
  We propose a scalable pre-training method named Policy Discriminative Learning (POLAR)<n>POLAR trains a reward model (RM) to discern identical policies and discriminate different ones.<n> Empirical results show that POLAR substantially outperforms traditional non-pre-trained methods.
  arXiv  Detail & Related papers  (2025-07-07T16:56:31Z)
• Adversarial Preference Learning for Robust LLM Alignment [24.217309343426297]
  Adversarial Preference Learning (APL) is an iterative adversarial training method incorporating three key innovations.<n>First, a direct harmfulness metric based on the model's intrinsic preference probabilities.<n>Second, a conditional generative attacker that synthesizes input-specific adversarial variations.
  arXiv  Detail & Related papers  (2025-05-30T09:02:07Z)
• Think, Prune, Train, Improve: Scaling Reasoning without Scaling Models [1.96238419451815]
  Large language models (LLMs) have demonstrated strong capabilities in programming and mathematical reasoning tasks, but are constrained by limited high-quality training data.<n>We introduce a scalable framework that iteratively fine-tunes models on their own reasoning traces, using ground-truth pruning to ensure high-quality training data.<n>This approach yields improved performance: on GSM8K, Gemma2-2B achieves a Pass@1 of 57.6% (from 41.9%), Gemma2-9B reaches 82%, matching LLaMA-3.1-70B, and LLaMA-3.1-70B attains 91%, even surpassing GPT-4o
  arXiv  Detail & Related papers  (2025-04-25T06:48:55Z)
• Exploring the Limit of Outcome Reward for Learning Mathematical Reasoning [65.2421542320293]
  Reasoning abilities are crucial components of general intelligence.<n>Recent advances by proprietary companies, such as o-series models of OpenAI, have made remarkable progress on reasoning tasks.<n>This paper proposes a new RL framework, termed OREAL, to pursue the performance limit that can be achieved through textbfOutcome textbfREwtextbfArd-based reinforcement textbfLearning for mathematical reasoning tasks.
  arXiv  Detail & Related papers  (2025-02-10T18:57:29Z)
• Kimi k1.5: Scaling Reinforcement Learning with LLMs [84.95584393629998]
  We report on the training practice of Kimi k1.5, our latest multi-modal language model trained with reinforcement learning.<n>Long context scaling and improved policy optimization methods are key ingredients of our approach.<n>Our system achieves state-of-the-art reasoning performance across multiple benchmarks and modalities.
  arXiv  Detail & Related papers  (2025-01-22T02:48:14Z)
• Tactile Grasp Refinement using Deep Reinforcement Learning and Analytic Grasp Stability Metrics [70.65363356763598]
  We show that analytic grasp stability metrics constitute powerful optimization objectives for reinforcement learning algorithms. We show that a combination of geometric and force-agnostic grasp stability metrics yields the highest average success rates of 95.4% for cuboids. In a second experiment, we show that grasp refinement algorithms trained with contact feedback perform up to 6.6% better than a baseline that receives no tactile information.
  arXiv  Detail & Related papers  (2021-09-23T09:20:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.