TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents

• URL: http://arxiv.org/abs/2606.05784v1
• Date: Thu, 04 Jun 2026 07:15:43 GMT
• Title: TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents
• Abstract summary: We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
• Score: 54.08846865906602
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We identify and formally characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents: its uniform broadcast of trajectory-level advantages to all tokens causes valuable tool-use steps in failing trajectories to be penalized no differently from valueless ones. We further empirically quantify the scale of this phenomenon. Over half of failing trajectories and failing tool-use actions exhibit correctable credit misassignment, demonstrating that the wasted training signal is both substantial and structurally exploitable. Building on this insight, we propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools: similar call parameters define equivalent information-acquisition actions and should therefore share comparable action credit. TAPO constructs counterfactual witnesses within the current training batch and compensates misassigned negative credit via confidence-gated conservative advantage correction. It requires no additional annotation, models, or sampling, and introduces negligible computational overhead. Across multiple multimodal search benchmarks, TAPO delivers consistent, plug-and-play improvements over strong baselines for three mainstream RL algorithms (GRPO, GSPO, and SAPO). Our code and models will be publicly released upon acceptance.

Related papers

• Neuro-symbolic PRM: Enhancing Scientific Reasoning via Structured Traces and Symbolic Verification [10.892972402027494]
  Intermediate reasoning steps are syntactically well-formed, mathematically executable, and unit-consistent, yet contextually ungrounded.<n>Current approaches either rely on formal verifiers that cannot assess semantic intent, or burden Process Reward Models (PRMs) with the dual task of checking both arithmetic and logic.<n>We propose a neuro-symbolic framework that cleanly decouples reasoning into two formal dimensions: Symbolic Validity ($V$) and Semantic Groundedness ($G$)
  arXiv  Detail & Related papers  (2026-08-26T19:10:12Z)
• IAPO: Influence-Aware Policy Optimization for Credit Assignment in Multi-Turn Service Agents [10.577620258047665]
  This work advances the understanding of credit assignment in multi-turn user interactions.<n>It provides a principled approach to training service agents from sparse outcome feedback.
  arXiv  Detail & Related papers  (2026-08-25T14:14:36Z)
• Agentic Security: A Systematization of Tools, Failure Modes, and Design Laws for LLM-Driven Penetration Testing [0.21990652930491858]
  Agentic security uses large-language-model (LLM) agents to plan, dispatch, and interpret security tools.<n>We systematize these failures through a hands-on evaluation of ten widely used static, dynamic, cloud, orchestration, and AI red-teaming tools.
  arXiv  Detail & Related papers  (2026-08-16T07:18:04Z)
• VerifyMAS: Hypothesis Verification for Failure Attribution in LLM Multi-Agent Systems [79.51005192758262]
  Large language model-driven multi-agent systems excel at complex tasks.<n>However, unreliable agents remain a key bottleneck to system-level reliability.<n>We propose a hypothesis verification framework for agent failure attribution.
  arXiv  Detail & Related papers  (2026-05-17T14:09:35Z)
• Interactive Critique-Revision Training for Reliable Structured LLM Generation [18.00222080273147]
  We propose DPA-GRPO, a paired-action training method for a two-player generator--verifier game with structured verifier interventions.<n>We analyze the unregularized game and show that positive probability on strictly lower-reward intervention or revision actions creates a profitable unilateral deviation.<n>Experiments on TaxCalc TY24 show that DPA-GRPO improves structured decision accuracy over zero-shot generation and generator-only RL baselines.
  arXiv  Detail & Related papers  (2026-05-08T17:00:38Z)
• LiveFMBench: Unveiling the Power and Limits of Agentic Workflows in Specification Generation [75.05397479715576]
  Large language models (LLMs) and agents have shown promising progress, but their true capabilities and failure modes remain unclear.<n>We present the first systematic and contamination-aware study of LLM- and agent-based formal specification generation for C programs.
  arXiv  Detail & Related papers  (2026-05-02T11:31:33Z)
• AgentV-RL: Scaling Reward Modeling with Agentic Verifier [63.55502685076245]
  Verifiers have been demonstrated to enhance LLM reasoning via test-time scaling (TTS)<n>We propose Agentic Verifier, a framework that transforms reward modeling into a multi-turn, tool-augmented deliberative process.<n>We show that Agentic Verifier yields consistent performance gains under both parallel and sequential TTS.
  arXiv  Detail & Related papers  (2026-04-17T12:27:36Z)
• Act Wisely: Cultivating Meta-Cognitive Tool Use in Agentic Multimodal Models [27.250148827297604]
  HDPO is a framework that reframes tool efficiency from a competing scalar objective to a strictly conditional one.<n>Our resulting model, Metis, reduces tool invocations by orders of magnitude while simultaneously elevating reasoning accuracy.
  arXiv  Detail & Related papers  (2026-04-09T17:59:57Z)
• Improving Search Agent with One Line of Code [68.58667107354253]
  Tool-based Agentic Reinforcement Learning (TARL) has emerged as a promising paradigm for training search agents to interact with external tools.<n>We propose textbfSearch textbfAgent textbfPolicy textbfOptimization (textbfSAPO), which stabilizes training via a conditional token-level KL constraint.
  arXiv  Detail & Related papers  (2026-03-10T04:07:39Z)
• DLLM Agent: See Farther, Run Faster [94.74432470237817]
  Diffusion large language models (DLLMs) have emerged as an alternative to autoregressive (AR) decoding with appealing efficiency and modeling properties.<n>We study this in a controlled setting by instantiatingDLLM and AR backbones within the same agent workflow.<n>We find thatDLLM Agents are on average over 30% faster end to end than AR agents, with some cases exceeding 8x speedup.
  arXiv  Detail & Related papers  (2026-02-07T09:01:18Z)
• CLEANER: Self-Purified Trajectories Boost Agentic Reinforcement Learning [4.765206163164323]
  CLEANER exploits intrinsic self-correction capabilities to eliminate error-contaminated context during data collection.<n>Similarity-Aware Adaptive Rollback mechanism autonomously constructs clean, purified trajectories.<n>Results show average accuracy gains of 6%, 3%, and 5% over baselines.
  arXiv  Detail & Related papers  (2026-01-21T16:14:30Z)
• Hard Negative Sample-Augmented DPO Post-Training for Small Language Models [4.425580048633862]
  We propose a lightweight and pragmatic post-training pipeline that targets structured errors under realistic compute budgets.<n>We introduce a compact MathVerifier that decomposes a candidate solution into a six-dimensional error profile and aggregates it into interpretable wrongness and absurdity scores.<n> Experiments show that verifier-guided, weighted DPO yields more targeted improvements than vanilla SFT and unweighted DPO.
  arXiv  Detail & Related papers  (2025-12-17T06:15:52Z)
• Diverse Transformer Decoding for Offline Reinforcement Learning Using Financial Algorithmic Approaches [4.364595470673757]
  Portfolio Beam Search (PBS) is a simple-yet-effective alternative to Beam Search (BS)<n>We develop an uncertainty-aware diversification mechanism, which we integrate into a sequential decoding algorithm at inference time.<n>We empirically demonstrate the effectiveness of PBS on the D4RL benchmark, where it achieves higher returns and significantly reduces outcome variability.
  arXiv  Detail & Related papers  (2025-02-13T15:51:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.