VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning

• URL: http://arxiv.org/abs/2608.28128v1
• Date: Fri, 28 Aug 2026 09:43:54 GMT
• Title: VICT: Verifier-Instrumented Credit Tracing for Long-Horizon LLM Agent Reinforcement Learning
• Abstract summary: VICT (VerifierInstrumented Credit Tracing) is a trainingtime interface that traces atoms back to actions through dependency-valid proof edges.<n>It improves substantially over outcome-only training and achieves strong performance alongside recent fine-grained credit methods.
• Score: 6.237984544487965
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fine-grained credit assignment is a central challenge in reinforcement learning for long horizon LLM agents. Standard objectives often train from programmatically verifiable terminal rewards by broadcasting each sparse outcome to every action in a trajectory. Existing methods typically seek finer credit from the rollout side, constructing auxiliary trajectory signals or additional comparisons to estimate action importance. Although useful, these approaches still treat the verifier that judged success as a scalar reward, discarding its internal task structure. Our key insight is that many verifiable tasks already encode the relevant checks inside their terminal verifier. We propose VICT (VerifierInstrumented Credit Tracing), a training-time interface that exposes executable or evidence backed atoms and traces them back to actions through dependency-valid proof edges. VICT redistributes group-relative advantage only along those edges, shifting credit assignment from rollout-side inference to verifierside tracing. It preserves the original terminal reward, abstains when evidence is incomplete or ambiguous, and changes only the training-time advantage tensor, requiring no learned critic, process labels, branch rollouts, or inference-time verifier access. On ALFWorld and WebShop, VICT improves substantially over outcome-only training and achieves strong performance alongside recent fine-grained credit methods; ablations rule out dense atom rewards, final-commit credit, temporal proximity, and sparsity as sufficient explanations.

Related papers

• Agentic Reinforcement Learning with Self-Distilled Reward Shaping [12.091871903119698]
  Agentic reinforcement learning enables LLM agents to learn through interaction, but sparse trajectory-level rewards reveal success without identifying which intermediate decisions deserve credit.<n>We introduce Agentic Reinforcement Learning with Self-Distilled Reward Shaping, a framework for constructing return-associated token-level credit for language agents.<n>Experiments show that ADRS consistently improves performance on long-horizon tasks, with gains persisting across RL backbones, reduced-data settings, unseen tasks, and extended training.
  arXiv  Detail & Related papers  (2026-08-04T06:56:47Z)
• TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents [51.523913302114266]
  TRACE (Turn-level Reward Assignment via Credit Estimation) is a dense credit-assignment method for agentic reinforcement learning.<n>It represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values.<n>It substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data.
  arXiv  Detail & Related papers  (2026-07-15T16:16:42Z)
• STAMP: Provenance-Guided Credit Assignment for Deep Search Agents [16.1299556246389]
  Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring.<n>We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation.<n>On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points.
  arXiv  Detail & Related papers  (2026-07-13T07:12:55Z)
• PACT: Privileged Trace Co-Training for Multi-Turn Tool-Use Agents [57.69188221227875]
  We propose a Privileged trAce Co-Training framework for multi-turn tool-use agents.<n>The key idea is to use expert traces only as training-time optimization signals rather than rollout-time hints.<n>We show that PACT consistently improves over strong SFT- and RL-based baselines.
  arXiv  Detail & Related papers  (2026-06-15T04:46:23Z)
• PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment [35.978805768172656]
  Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning.<n>We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards.
  arXiv  Detail & Related papers  (2026-06-08T11:20:58Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning [10.87972575497941]
  Chain-of-thought (CoT) reasoning is useful for monitoring language models.<n>Models can rely on prompt-to-answer shortcuts that bypass the CoT.<n>We study CoT faithfulness through a structural information-flow perspective.
  arXiv  Detail & Related papers  (2026-05-22T23:37:29Z)
• Verifiable Process Rewards for Agentic Reasoning [21.195739597726142]
  Reinforcement learning from verifiable rewards (RLVR) has improved the reasoning abilities of large language models (LLMs)<n>Most existing approaches rely on sparse outcome-level feedback.<n>We propose Verifiable Process Rewards (VPR), a framework that converts such oracles into dense turn-level supervision for reinforcement learning.
  arXiv  Detail & Related papers  (2026-05-11T10:30:53Z)
• Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution [79.98699884805636]
  Reasoning Execution by Multiple Listeners (REMUL) is a multi-party reinforcement learning approach.<n>REMUL builds on the hypothesis that reasoning traces which other parties can follow will be more faithful.<n>Speakers are rewarded for producing reasoning that is clear to listeners.
  arXiv  Detail & Related papers  (2026-02-18T02:55:55Z)
• From Verifiable Dot to Reward Chain: Harnessing Verifiable Reference-based Rewards for Reinforcement Learning of Open-ended Generation [52.62655622099456]
  We propose reinforcement learning with verifiable reference-based rewards (RLVRR)<n>Instead of checking the final answer, RLVRR extracts an ordered linguistic signal from high-quality references (i.e., reward chain)<n>In this way, RLVRR decomposes rewards into two dimensions: content, which preserves deterministic core concepts, and style, which evaluates adherence to stylistic properties.
  arXiv  Detail & Related papers  (2026-01-26T14:39:58Z)
• Look As You Think: Unifying Reasoning and Visual Evidence Attribution for Verifiable Document RAG via Reinforcement Learning [55.232400251303794]
  Look As You Think (LAT) is a reinforcement learning framework that trains models to produce verifiable reasoning paths with consistent attribution.<n>LAT consistently improves the vanilla model in both single- and multi-image settings, yielding average gains of 8.23% in soft exact match (EM) and 47.0% in IoU@0.5.
  arXiv  Detail & Related papers  (2025-11-15T02:50:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.