PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents

• URL: http://arxiv.org/abs/2606.16215v1
• Date: Mon, 15 Jun 2026 04:46:23 GMT
• Title: PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents
• Abstract summary: We propose a Privileged trAce Co-Training framework for multi-turn tool-use agents.<n>The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints.<n>We show that PACT consistently improves over strong SFT- and RL-based baselines.
• Score: 57.69188221227875
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-turn tool-use agents must reason, call tools, and adapt to observations across several interaction turns. Post-training such agents is challenging, as reinforcement learning often suffers from sparse rewards and weak credit assignment despite matching the prompt-only inference setting, while supervised fine-tuning on expert traces provides dense process supervision but can over-constrain the model to fixed trajectories. To tackle this, we propose PACT, a Privileged trAce Co-Training framework for multi-turn tool-use agents. The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints. PACT keeps rollout generation prompt-only, then uses expert traces to guide optimization through two complementary signals: a trace-conditioned RL surrogate that evaluates prompt-only rollouts under expert-trace context, and a component-aware SFT loss that supervises reasoning prefixes and tool-calls with annealed strength. To reduce over-reliance on the training-only trace context, PACT further introduces a prompt-only anchoring. We also provide a latent-trace view that connects the two trace-based objectives and explains how expert traces can guide optimization without being used during rollout generation. Experiments on FTRL, BFCL, and ToolHop show that PACT consistently improves over strong SFT- and RL-based baselines, highlighting the value of privileged trace co-training for multi-turn tool-use learning.

Related papers

• Eliciting Self-Verification in Multimodal Reasoning Agents with Reinforcement Learning [9.939107031203838]
  Self-Verification via Reinforcement Learning (SVRL) trains multimodal agents to verify and filter retrieved evidence within their own reasoning traces.<n>SVRL also introduces a search-aware penalty that discourages unnecessary tool calls and a query-diversity reward that encourages diverse, well-formed search queries.<n>Overall, SVRL narrows the gap between compact agents and much larger proprietary models while requiring substantially lower training and inference cost.
  arXiv  Detail & Related papers  (2026-09-07T22:15:47Z)
• Revisiting Complete Reasoning Traces for Post-Training [61.76889440384448]
  Large language models (LLMs) are often post-trained on pre-collected reasoning trajectories to improve their reasoning capability.<n>We show that full trajectories provide only limited benefit, while partial trajectories are effective even under heavy truncation.
  arXiv  Detail & Related papers  (2026-09-07T06:46:17Z)
• HiDiffTIR: Hierarchical Difficulty-Aware Policy Optimization for Multi-Turn Tool-Integrated Reasoning [95.29291773706224]
  Tool-Integrated Reasoning is a capability for LLM agents to solve complex tasks by interacting with external tools iteratively.<n>Existing approaches typically assign uniform trajectory-level advantages and treat all correct tool calls equally.<n>This can lead to imprecise learning signals that do not adequately distinguish between trivial and challenging tool-use patterns.
  arXiv  Detail & Related papers  (2026-08-22T09:14:15Z)
• Why Multi-Step Tool-Use Reinforcement Learning Collapses and How Supervisory Signals Fix It [23.538941671680806]
  Tool use enables large language models to perform complex tasks.<n>Recent agentic reinforcement learning methods show promise for enhancing model capabilities.<n>Some models exhibit catastrophic collapse, where performance abruptly drops and tool-invocation structures fail.
  arXiv  Detail & Related papers  (2026-06-24T16:55:56Z)
• Self-Play Enhancement via Advantage-Weighted Refinement in Online Federated LLM Fine-Tuning with Real-Time Feedback [37.79391516435725]
  We introduce SPEAR, an efficient online learning algorithm for federated feedback fine-tuning.<n>We validate SPEAR across various benchmark datasets, demonstrating its superior performance in comparison to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-08T16:35:42Z)
• Waking Up Blind: Cold-Start Optimization of Supervision-Free Agentic Trajectories for Grounded Visual Perception [18.565166776233223]
  Small Vision-Language Models (SVLMs) are efficient task controllers but often suffer from visual brittleness and poor tool orchestration.<n>We propose Self-supervised Perception Enabled by Cascaded Tool Rollout Alignment (SPECTRA)<n>SPECTRA bootstraps agentic capabilities via Coldstart Reinforcement Learning for SVLMs.
  arXiv  Detail & Related papers  (2026-04-19T15:06:30Z)
• Multimodal Latent Reasoning via Predictive Embeddings [43.40267514669565]
  Pearl is a framework that learns from expert tool-use trajectories entirely in the latent space.<n>Pearl is model-agnostic, simple to train, and naturally supports trajectories with multiple tool calls.
  arXiv  Detail & Related papers  (2026-04-09T10:27:32Z)
• In-Context Reinforcement Learning for Tool Use in Large Language Models [68.66653829365187]
  Large language models (LLMs) exhibit strong reasoning abilities, but their performance on complex tasks is often constrained by their internal knowledge.<n>We propose In-Context Reinforcement Learning (ICRL), an RL-only framework that eliminates the need for supervised fine-tuning.<n>ICRL achieves state-of-the-art performance, demonstrating its effectiveness as a scalable, data-efficient alternative to traditional SFT-based pipelines.
  arXiv  Detail & Related papers  (2026-03-09T08:06:18Z)
• Learning in Context, Guided by Choice: A Reward-Free Paradigm for Reinforcement Learning with Transformers [55.33468902405567]
  We propose a new learning paradigm, In-Context Preference-based Reinforcement Learning (ICPRL), in which both pretraining and deployment rely solely on preference feedback.<n>ICPRL enables strong in-context generalization to unseen tasks, achieving performance comparable to ICRL methods trained with full reward supervision.
  arXiv  Detail & Related papers  (2026-02-09T03:42:16Z)
• SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning [83.66308307152808]
  We propose StAbilized Mixture-of-Experts (SAME) for Multimodal Continual Instruction Tuning (MCIT)<n>SAME stabilizes expert selection by decomposing routing dynamics into subspaces and updating only task-relevant directions.<n>It also introduces adaptive expert activation to freeze selected experts during training, reducing redundant and cross-task interference.
  arXiv  Detail & Related papers  (2026-02-02T11:47:06Z)
• Just-in-time Episodic Feedback Hinter: Leveraging Offline Knowledge to Improve LLM Agents Adaptation [77.90555621662345]
  We present JEF Hinter, an agentic system that distills offline traces into compact, context-aware hints.<n>A zooming mechanism highlights decisive steps in long trajectories, capturing both strategies and pitfalls.<n>Experiments on MiniWoB++, WorkArena-L1, and WebArena-Lite show that JEF Hinter consistently outperforms strong baselines.
  arXiv  Detail & Related papers  (2025-10-05T21:34:42Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.