Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry

• URL: http://arxiv.org/abs/2607.03702v1
• Date: Sat, 04 Jul 2026 04:45:05 GMT
• Title: Agent Reinforcement Learning via Pivotal-Aware Self-Feedback Retry
• Abstract summary: We propose PivoARL, a self-feedback retry framework for experience exploitation in large language model (LLM) agents.<n>PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state.<n>We show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5% over MetaRL.
• Score: 14.02751159295063
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language model (LLM) agents have shown strong decision-making capabilities in long-horizon interactive tasks, yet they still struggle to effectively leverage failed trajectories: full retries incur high interaction costs, while experience retrieval tends to dilute critical experience signals. To address this, we propose PivoARL, a self-feedback retry framework for experience exploitation in LLM agents. PivoARL identifies the pivotal erroneous turn through structured reflection and performs local retry only from the corresponding pivotal state, thereby reusing the correct prefix and reducing redundant interactions. From an information-gain perspective, we further show that pivotal retry concentrates useful experience signals near the error boundary, mitigating the signal dilution caused by state-agnostic experience utilization. Based on this insight, we design a pivotal-aware credit assignment mechanism that rewards correct prefixes while isolating erroneous suffixes, and optimize reflection quality through implicit reflection returns. We conduct a systematic evaluation on 4 agent tasks and 7 search-based QA benchmarks. Results show that PivoARL achieves significant improvements on Pass@2/3 across all tasks, with an average gain of about 11.5\% over MetaRL. Moreover, benefiting from contrastive preference signals induced by pivotal turns, PivoARL also consistently improves Pass@1 on over 80\% of the tasks. On Minesweeper environment, PivoARL improves over GiGPO by more than 45\% and reduces interaction turns by about 42\% on average compared with full-retry methods. Code is available at https://github.com/yuki-younai/PivoARL.

Related papers

• Finding Where the Buck Stops: An Automated Failure Attribution-Based Reflection Framework for Multi-Agent Collaboration [14.41671360589919]
  DoCtOR (Diagnose-then-Correct PPO-enhanced Reflection) is a novel reflection framework that enhances multi-agent collaboration.<n>DoCtOR achieves 22%, 26%, and 27% improvements over initial success rates on HotPotQA, ChartQAPro, and Mind2Web datasets.
  arXiv  Detail & Related papers  (2026-08-28T12:26:03Z)
• A-SR: Self-Evolving Agentic LLMs for Symbolic Regression via Hierarchical Coordination [51.06539604709775]
  We propose A-SR, a self-evolving agentic framework that shifts the control unit from expression edits to role-conditioned evidence views.<n>A-SR coordinates formula discovery through routing among coordination protocols, an online evaluator-reward role policy, and state-routed process memory.
  arXiv  Detail & Related papers  (2026-08-05T14:01:10Z)
• ReGRPO: Reflection-Augmented Policy Optimization for Tool-Using Agents [52.531681772560724]
  Supervised fine-tuning (SFT) is built mostly on successful trajectories and offers little signal for recovery after tool failures.<n>We introduce ReGRPO, a framework that learns reflection-guided correction in tool-using agents.<n> Experiments on GTA and GAIA show that, under the same backbone and tool suite, ReGRPO consistently outperforms strong open-source baselines.
  arXiv  Detail & Related papers  (2026-06-30T09:19:38Z)
• Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL [12.234169944475537]
  RefGRPO is a simple yet effective fix that augments standard RL algorithms with two key ingredients.<n>A free calibration bonus computed by contrasting the agent's own reflection with the actual outcome.<n>The resulting reflection turns the agent into its own verifier grounded in environment feedback.
  arXiv  Detail & Related papers  (2026-06-12T07:47:15Z)
• ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents [48.80766702702854]
  LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised.<n>This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient.<n>We propose ARBOR (Adaptive Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries.
  arXiv  Detail & Related papers  (2026-06-02T06:58:54Z)
• What and When to Distill: Selective Hindsight Distillation for Multi-Turn Agents [70.6980022118038]
  Reinforcement learning can train LLM agents from sparse task rewards, but long-horizon credit assignment remains challenging.<n>Existing methods rely on trajectory-level rewards or proxy signals, without fully leveraging per-step environmental feedback.<n>We introduce SERL, a selective environment-reweighted learning framework.
  arXiv  Detail & Related papers  (2026-05-19T07:00:55Z)
• Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation [50.696688705287755]
  We propose Mutual Information Self-Evaluation to overcome the sparse reward challenge in reinforcement learning.<n>MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals.<n>We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy.
  arXiv  Detail & Related papers  (2026-04-13T15:18:51Z)
• Internalizing Agency from Reflective Experience [20.650609947690196]
  LEAFE is a framework that internalizes recovery agency from reflective experience.<n>It consistently improves Pass@1 over the base model and achieves higher Pass@k than outcome-driven baselines.
  arXiv  Detail & Related papers  (2026-03-17T17:50:47Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.