Retrospective Progress-Aware Self-Refinement for LLM Agent Training

• URL: http://arxiv.org/abs/2606.14302v1
• Date: Fri, 12 Jun 2026 09:38:47 GMT
• Title: Retrospective Progress-Aware Self-Refinement for LLM Agent Training
• Abstract summary: RePro is a framework that trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm.<n>RePro enhances the Qwen family's performance, with up to $12%$ absolute success rate gains.
• Score: 55.2682415036601
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-based agents trained with reinforcement learning optimize step-wise action prediction but lack metacognitive awareness of task progress, inducing a gap that hinders long-horizon scaling. A pilot study reveals that online progress prompting hurts performance while retrospective demonstrations help, yet this capability cannot emerge from outcome-reward training alone. We present RePro, Retrospective Progress-Aware Training, a framework that trains agents to self-generate progress signals via a forward-then-reflect rollout paradigm: the agent executes actions online, then retrospectively reassesses its step-wise progress given the completed trajectory and known outcome. RePro initializes with a Retrospection Warmup that teaches reflection format from minimal external demonstrations, then further trains through RePro-PO with a composite reward that produces self-generated signals without continuous external supervision. Experiments on WebShop, ALFWorld, and Sokoban show that RePro enhances the Qwen family's performance, with up to $12\%$ absolute success rate gains.

Related papers

• Harnessing LLM Agents with Skill Programs [58.356514745548026]
  We introduce HASP, a new framework that upgrades skills into executable Program Functions (PFs)<n>PFs act as executable guardrails that activate on failure-prone states and modify the next action or inject corrective context.<n> HASP drives substantial gains compared to both training-free and training-based methods on web-search, math reasoning, and coding tasks.
  arXiv  Detail & Related papers  (2026-05-18T01:35:11Z)
• Learning with Rare Success but Rich Feedback via Reflection-Enhanced Self-Distillation [71.16738415436458]
  We introduce Reflection-Enhanced Self-Distillation (RESD), a framework that transforms raw failure feedback into an active source of corrective supervision.<n>RESD interprets failed trajectories by generating retrospective reflections to diagnose local errors, and curates a persistent global playbook.<n> Empirical evaluations on multiple continual learning tasks demonstrate that RESD substantially outperforms standard self-distillation baselines.
  arXiv  Detail & Related papers  (2026-05-12T20:46:05Z)
• RetroAgent: From Solving to Evolving via Retrospective Dual Intrinsic Feedback [54.39884046754265]
  RetroAgent is an online RL framework that enables agents to master complex interactive environments.<n>Our experiments show that RetroAgent achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-03-09T16:23:33Z)
• ProgAgent:A Continual RL Agent with Progress-Aware Rewards [0.07646713951724009]
  ProgAgent is a continual reinforcement learning agent that unifies progress-aware reward learning with a JAX-native system architecture.<n>It derives dense, shaped rewards from unlabeled expert videos through a perceptual model that estimates task progress across initial, current, and goal observations.<n>ProgAgent supports massively parallel rollouts and fully differentiable updates, rendering a sophisticated unified objective feasible.
  arXiv  Detail & Related papers  (2026-03-08T19:58:07Z)
• Learning from Trials and Errors: Reflective Test-Time Planning for Embodied LLMs [63.88783817420284]
  Embodied robots cannot reflect on what went wrong or why, turning deployment into a sequence of independent trials.<n>We introduce Reflective Test-Time Planning, which integrates two modes of reflection: textitreflection-in-action and textitreflection-on-action<n>We also include retrospective reflection, allowing the agent to re-evaluate earlier decisions and perform model updates with hindsight.
  arXiv  Detail & Related papers  (2026-02-24T18:55:18Z)
• Experiential Reinforcement Learning [22.545003569634982]
  Experiential Reinforcement Learning (ERL) is a training paradigm that embeds an explicit experience-reflection-consolidation loop into the reinforcement learning process.<n>ERL consistently improves learning efficiency and final performance over strong reinforcement learning baselines.<n>These results suggest that integrating explicit self-reflection into policy training provides a practical mechanism for transforming feedback into durable behavioral improvement.
  arXiv  Detail & Related papers  (2026-02-15T01:23:48Z)
• Explore Data Left Behind in Reinforcement Learning for Reasoning Language Models [61.78513830395669]
  Reinforcement Learning with Verifiable Rewards (RLVR) has emerged as an effective approach for improving the reasoning abilities of large language models (LLMs)<n>As models train longer and scale larger, more training prompts become residual prompts, those with zero variance rewards that provide no training signal.<n>We propose the Explore Residual Prompts in Policy Optimization framework, which encourages exploration on residual prompts and reactivates their training signals.
  arXiv  Detail & Related papers  (2025-11-06T20:40:27Z)
• Learn the Ropes, Then Trust the Wins: Self-imitation with Progressive Exploration for Agentic Reinforcement Learning [41.90621652673528]
  We propose SPEAR, a curriculum-based self-imitation learning (SIL) recipe for training agentic LLMs.<n>Specifically, our approach incorporates a curriculum to manage the exploration process, utilizing intrinsic rewards to foster skill-level exploration.<n>To further stabilize training, we recalibrate the advantages of experiences in the replay buffer to address the potential policy drift.
  arXiv  Detail & Related papers  (2025-09-26T17:20:38Z)
• PROGRESSOR: A Perceptually Guided Reward Estimator with Self-Supervised Online Refinement [16.768912344111946]
  We present PROGRESSOR, a framework that learns a task-agnostic reward function from videos.<n>We show that PROGRESSOR enables robots to learn complex behaviors without any external supervision.
  arXiv  Detail & Related papers  (2024-11-26T04:17:51Z)
• SELF: Self-Evolution with Language Feedback [68.6673019284853]
  'SELF' (Self-Evolution with Language Feedback) is a novel approach to advance large language models. It enables LLMs to self-improve through self-reflection, akin to human learning processes. Our experiments in mathematics and general tasks demonstrate that SELF can enhance the capabilities of LLMs without human intervention.
  arXiv  Detail & Related papers  (2023-10-01T00:52:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.