LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning

• URL: http://arxiv.org/abs/2607.13501v2
• Date: Thu, 16 Jul 2026 15:39:38 GMT
• Title: LOTAPO: Leave-One-Turn Attribution for Self-Generated Process Rewards in Multi-Turn Search Reasoning
• Abstract summary: LOTAPO is a self-generated process-supervision method based on backward leave-one-turn attribution.<n>For each search turn, LOTAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder.<n>This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions.
• Score: 12.013691798765594
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning for multi-turn search reasoning typically relies on terminal outcome rewards, which cannot distinguish useful, redundant, and harmful intermediate interactions. We propose LOTAPO , a self-generated process-supervision method based on backward leave-one-turn attribution. For each search turn, LOTAPO replaces the turn and its retrieval observation with a fixed [DELETE] placeholder and measures the resulting change in the current policy's mean log-likelihood of the gold answer. This Answer-Likelihood Gain estimates the turn's contribution while preserving all downstream interactions, allowing early evidence to be evaluated in the complete reasoning context. LOTAPO further applies sign-consistency gating, retaining only normalized process advantages whose directions agree with their raw attribution scores. The method requires no additional reward model, teacher, verifier, or LLM-as-a-Judge. Across seven knowledge-intensive question-answering datasets with local retrieval, LOTAPO achieves an average exact-match score of 0.326, outperforming the strongest step-reward baseline, IGPO, by 0.053. Ablations show complementary benefits from backward attribution and sign-consistency gating, demonstrating that policy-derived retrospective attribution can provide effective process supervision for multi-turn search agents.

Related papers

• ARISE-RL: Agentic Rubric-Grounded Iterative Self-Evolution with Reinforcement Learning [74.19730275669227]
  ARISE-RL is a novel full-cycle self-evolution framework for training open-ended agents.<n>We present Reward-Gated Self-Evolution Distillation (RG-SED), which distills a memory-augmented variant of the same policy back into itself.<n>We also present ECR-Bench, an expert-calibrated rubric benchmark suite covering single-tool deep research and multi-tool travel planning.
  arXiv  Detail & Related papers  (2026-09-01T10:54:13Z)
• Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization [13.354353071601379]
  We introduce TASPO, which converts privileged supervision into outcome-grounded action credit.<n>Across three agentic benchmarks, TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks.
  arXiv  Detail & Related papers  (2026-08-31T16:51:50Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning [10.917273110894469]
  Agentic reinforcement learning requires assigning credit to environment-facing actions.<n>Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens.<n>We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit.
  arXiv  Detail & Related papers  (2026-06-30T17:48:07Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Reward Bias Substitution: Single-Axis Bias Mitigations Redirect Optimization Pressure [23.021647176206972]
  Single-axis mitigations of reward-model biases can rotate optimization pressure onto correlated proxies rather than eliminate it.<n>The failure is enabled by a measurement-versus-optimization gap between audit and policy-induced distributions.<n>We formalize mitigation outcomes into a regime taxonomy and prove that successful mitigation, bias substitution, and overcorrection produce identical observables.
  arXiv  Detail & Related papers  (2026-05-27T05:40:22Z)
• RICE-PO: Turning Retrieval Interactions into Credit Signals for Reasoning Agents [37.33108522642245]
  We propose RICE-PO, a critic-free policy optimization framework that converts retrieval interactions into localized learning signals.<n>On BRIGHT and BEIR, RICE-PO consistently outperforms prompt-based agents and group-based RL baselines under the same retriever setting.
  arXiv  Detail & Related papers  (2026-05-25T21:56:29Z)
• The Reciprocity Gradient [52.35929743862925]
  Communication is fundamental to sustaining reciprocity and cooperation in strategic interactions.<n>We identify and formulate the influence attribution problem as the central optimization difficulty inherent in such dynamics for a learning agent.<n>To address this, we introduce the reciprocity gradient, which explicitly backpropagates reward gradients through private estimators of opponents' policies trained from public observations.
  arXiv  Detail & Related papers  (2026-05-08T16:29:52Z)
• Self-Induced Outcome Potential: Turn-Level Credit Assignment for Agents without Verifiers [26.97849381770806]
  Self-Induced Outcome Potential treats semantic clusters of final answers as latent future outcome states for potential-based turn-level credit assignment.<n>We formalize the framework, characterize its supervised gold-answer limit, and show that SIOP improves average performance over verifier-free outcome-level baselines.
  arXiv  Detail & Related papers  (2026-05-06T14:38:48Z)
• Enhancing LLM-based Search Agents via Contribution Weighted Group Relative Policy Optimization [47.7937991619078]
  We propose Contribution-Weighted GRPO, a framework that integrates process supervision into group relative policy optimization.<n>CW-GRPO employs an LLM judge to assess the retrieval utility and reasoning correctness at each search round, producing per-round contribution scores.<n> Experiments on multiple knowledge-intensive benchmarks show that CW-GRPO outperforms standard GRPO by 5.0% on Qwen3-8B and 6.3% on Qwen3-1.7B.
  arXiv  Detail & Related papers  (2026-04-15T17:37:59Z)
• Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation [50.696688705287755]
  We propose Mutual Information Self-Evaluation to overcome the sparse reward challenge in reinforcement learning.<n>MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals.<n>We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy.
  arXiv  Detail & Related papers  (2026-04-13T15:18:51Z)
• Evidence-Augmented Policy Optimization with Reward Co-Evolution for Long-Context Reasoning [52.144281362465996]
  We propose EAPO (Evidence-Augmented Policy Optimization) to apply Reinforcement Learning to long-context scenarios.<n>We first establish the Evidence-Augmented Reasoning paradigm, validating via Tree-Structured Evidence Sampling.<n>We then introduce a specialized RL algorithm where a reward model computes a Group-Relative Evidence Reward.<n>To sustain accurate supervision throughout training, we further incorporate an Adaptive Reward-Policy Co-Evolution mechanism.
  arXiv  Detail & Related papers  (2026-01-15T11:40:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.