Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning

• URL: http://arxiv.org/abs/2606.23112v1
• Date: Mon, 22 Jun 2026 09:56:44 GMT
• Title: Self-Evolution for Multi-Turn Tool-Calling Agents via Divergence-Point Preference Learning
• Abstract summary: Multi-turn tool-using agents must coordinate long-horizon tool sequences while tracking dialogue state and policy constraints.<n>For within-benchmark self-improvement, ToolGraph combines schema-derived topology, transition weights estimated from successful rollouts, and history-aware controls for write prerequisites and repeated-search loops.<n>We then construct 161 preference pairs by locating divergence points via state-based matching and prefix-based alignment, filtered through action-correctness annotations, and train DPO under the same ToolGraph context used at inference.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-turn tool-using agents must coordinate long-horizon tool sequences while tracking dialogue state and policy constraints. Existing approaches often separate inference-time orchestration from parameter-level learning, leaving tool selection weakly structured and preference updates vulnerable to train--deployment prompt mismatch. For within-benchmark self-improvement, ToolGraph combines schema-derived topology, transition weights estimated from successful rollouts, and history-aware controls for write prerequisites and repeated-search loops. We then construct 161 preference pairs by locating divergence points via state-based matching and prefix-based alignment, filtered through action-correctness annotations, and train DPO under the same ToolGraph context used at inference. Across 375 tau2-bench tasks, ToolGraph raises the weighted average reward from 0.304 to 0.338 (+11.2% relative), while ToolGraph+DPO reaches 0.355 (+16.8% over the baseline), with the DPO gain concentrated in airline and retail. Fine-grained diagnostics further show that roughly half of telecom trajectories exhaust the step budget before action execution and that chosen reward positivity is the most useful checkpoint signal across our 16 evaluated DPO configurations.

Related papers

• Amplitude-Only FFN Intervention for Tool-Structured LLM Inference Method: Gated Evaluation Protocol, and Cross-Model Empirical Results [10.526208975530054]
  We study inference-time feed-forward network (FFN) intervention for improving structured outputs without retraining model weights.<n>Amplitude Gating (AG) is a non-destructive alternative that preserves pretrained FFN weight directions and modulates only activation magnitudes during generation.<n>These results identify tool-structured inference as the most credible first target for safe FFN-level inference optimization.
  arXiv  Detail & Related papers  (2026-07-13T07:28:55Z)
• ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents [52.531681772560724]
  Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures.<n>We introduce ReGRPO, a framework that learns reflection-guided correction in tool-using agents.<n> Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:19:38Z)
• CRAFT: Learn the Schema, Execute the Plan [1.4711904351495848]
  We present CRAFT, a two-stage post-training recipe for schema-grounded coding agents.<n>First, schema-stripped PLAN supervised fine-tuning learns domain-structured plans and executable behaviors from validated trajectories.<n>Second, execution-shaped reinforcement learning aligns the policy for tool selection, code quality, plan-code consistency, and recovery from failed executions.
  arXiv  Detail & Related papers  (2026-06-24T00:12:03Z)
• HyperTool: Beyond Step-Wise Tool Calls for Tool-Augmented Agents [52.33697136670754]
  We introduce textbfHyperTool, a unified executable MCP-style tool interface that changes the model-visible unit of tool execution.<n>A model invokes HyperTool with a code block that can call existing tools through their original schemas, manipulate returned values, and pass intermediate results locally.<n>On MCP-Universe, HyperTool improves average accuracy from 15.69% to 35.29% on Qwen3-32B and from 9.93% to 33.33% on Qwen3-8B, and surpass GPT-OSS and Kimi-k2.5 on average
  arXiv  Detail & Related papers  (2026-06-11T17:56:36Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• ToolGate: Token-Efficient Pre-Call Control for Tool-Augmented Vision-Language Agents [9.793491545168575]
  ToolGate predicts execute/skip decisions from trajectory text and simple structural features.<n>Across two Qwen3-VL backbones, ToolGate reduces token cost to 64-69% of the unrestricted ReAct baseline.
  arXiv  Detail & Related papers  (2026-06-02T02:44:27Z)
• Agent Explorative Policy Optimization for Multimodal Agentic Reasoning [97.64835302176056]
  Agentic reasoning interleaves two behaviors with a structural asymmetry: thinking and tool use.<n>Under standard RL recipes like GRPO, the gap manifests as two diagnostic symptoms during training.<n>We propose AXPO (Agent eXplorative Policy Optimization) for each all-wrong tool-using subgroup.
  arXiv  Detail & Related papers  (2026-05-27T17:36:39Z)
• ParaVT: Taming the Tool Prior Paradox for Parallel Tool Use in Agentic Video Reinforcement Learning [91.51460129144654]
  We introduce ParaVT, the first multi-agent end-to-end RL-trained framework for Parallel Video Tool calling.<n>We show that ParaVT improves over the Qwen3-VL baseline by +7.9% on average, with PARA-GRPO lifting training-time format compliance from 0.13 to 0.64.
  arXiv  Detail & Related papers  (2026-05-19T18:01:26Z)
• When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agents [32.54888729689753]
  We study failures as a sim-to-real gap in the tool-use partially observable Markov decision process (POMDP)<n>We propose ToolRL-DR, a domain-randomization reinforcement learning (RL) recipe that trains a tool-use agent on perturbation-augmented trajectories.<n>On a 3B backbone, ToolRL-DR-Full retains roughly three-quarters of clean accuracy and reaches an aggregate accuracy comparable to open-source 14B function-calling baselines.
  arXiv  Detail & Related papers  (2026-05-12T10:40:28Z)
• Beyond Pairs: Your Language Model is Secretly Optimizing a Preference Graph [17.030746750590758]
  We propose a principled generalization of DPO that operates over directed acyclic preference graphs induced by rollout rankings.<n>GraphDPO encodes dominance relations as edges and optimize a graph-structured Plackett--Luce-inspired objective.<n> Experiments on reasoning and program tasks demonstrate superior performance, suggesting that graph-structured preference modeling is a scalable and robust alternative to pairwise and listwise alignment objectives.
  arXiv  Detail & Related papers  (2026-05-08T17:26:14Z)
• UniToolCall: Unifying Tool-Use Representation, Data, and Evaluation for LLM Agents [22.52508596251479]
  We present UniToolCall, a unified framework for tool learning that standardizes the entire pipeline from construction and dataset generation to evaluation.<n>We convert 7 public benchmarks into a unified Query-Action--Observation-Answer (QAOA) representation with fine-grained evaluation at the function-call, turn, and conversation levels.<n>Under the Anchoror-heavy Hybrid-20 setting, achieves 93.0% single-turn Strict Precision, outperforming commercial models including GPT, Gemini, and Claude.
  arXiv  Detail & Related papers  (2026-04-13T14:43:47Z)
• In-the-Flow Agentic System Optimization for Effective Planning and Tool Use [73.72524040856052]
  AgentFlow is a trainable, in-the-flow agentic framework that coordinates four modules (planner, executor, verifier, generator) through an evolving memory.<n>Flow-GRPO tackles long-horizon, sparse-reward credit assignment by converting multi-turn optimization into a sequence of tractable single-turn policy updates.<n>AgentFlow with a 7B-scale backbone outperforms top-performing baselines with average accuracy gains of 14.9% on search, 14.0% on agentic, 14.5% on mathematical, and 4.1% on scientific tasks.
  arXiv  Detail & Related papers  (2025-10-07T05:32:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.