Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning

• URL: http://arxiv.org/abs/2608.13026v1
• Date: Thu, 13 Aug 2026 09:54:14 GMT
• Title: Temporal GRPO: Beyond Trajectory-Level Credit in Vision-Language-Action Reinforcement Learning
• Abstract summary: Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback.<n>On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons.
• Score: 27.507258147328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Outcome-driven reinforcement learning offers a scalable way to post-train vision-language-action (VLA) policies from sparse task-success feedback. In common GRPO-based VLA post-training, one rollout-level advantage is applied to every action in the trajectory. A rollout that completes several valid stages but fails later can therefore penalize the actions that produced its earlier progress. We call this trajectory-level credit aliasing. Temporal GRPO addresses this problem by constructing detectable task stages, aligning each rollout with stage-specific action intervals, and comparing only rollouts that have entered the same stage. The resulting stage advantages are applied to their corresponding intervals in a single policy update. On RoboTwin 2.0, Temporal GRPO improves task success and sample efficiency, with consistent gains across task horizons. Controlled updates on LIBERO-Long preserve shared prerequisite stages and concentrate improvement at the first stage where rollout outcomes diverge.

Related papers

• PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulation [17.66354304197193]
  Policy improvement requires step-level credit signals to distinguish behaviors that advance the task from those that stall or regress.<n>We present PACE, a credit-assignment framework for post-training on long-horizon manipulation.
  arXiv  Detail & Related papers  (2026-08-15T04:25:16Z)
• Progress-conditioned Group Policy Optimization for Long-Horizon Agentic Tasks [17.832722956008656]
  Group-based policy optimization has been increasingly used to train large language model (LLM) agents from sparse outcome rewards.<n>We propose Progress-conditioned Group Policy Optimization (ProGPO), which uses first-visit observation coverage only when all samples in a group receive zero outcome reward.<n>Experiments on two challenging agentic benchmarks, ALFWorld and WebShop with Qwen2.5-1.5/7B-Instruct, show that ProGPO consistently improves over group-based baselines.
  arXiv  Detail & Related papers  (2026-07-22T04:01:56Z)
• Group-Graph Policy Optimization for Long-Horizon Agentic Reinforcement Learning [85.02566758103008]
  Group-Graph Policy Optimization (G2PO) is a novel group-based Reinforcement Learning algorithm tailored for multi-turn agentic tasks.<n>G2PO substantially outperforms state-of-the-art prompt-based and RL baselines, achieving remarkable success rate improvements of up to 22.2% over GRPO.
  arXiv  Detail & Related papers  (2026-06-22T08:12:47Z)
• ProgVLA: Progress-Aware Robot Manipulation Skill Learning [13.001629637153583]
  We present ProgVLA, a compact vision-language-action (VLA) model for reliable robot manipulation under tight compute and memory budgets.<n>A multi-modal encoder with a two-stage Perceiver resampling scheme compresses variable-length visual, language, and proprioceptive streams into a fixed set of control-ready context tokens.<n>An auxiliary set of progress heads is trained with offline reinforcement learning objectives to jointly learn critics over normalized remaining-horizon targets.
  arXiv  Detail & Related papers  (2026-05-27T09:44:46Z)
• PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations [66.94988600664574]
  We present textbfPRTS (textbfPrimitive textbfReasoning and textbfTasking textbfSystem), a VLA foundation model that reformulates pretraining through Goal-Conditioned Reinforcement Learning.
  arXiv  Detail & Related papers  (2026-04-30T06:14:02Z)
• Jump-Start Reinforcement Learning with Vision-Language-Action Regularization [1.2599533416395767]
  Reinforcement learning (RL) enables high-frequency, closed-loop control for robotic manipulation.<n>Current limitations hinder their direct use in fast and precise manipulation.<n>We propose Vision-Language-Action Jump-Starting (VLAJS) to improve exploration and learning efficiency.
  arXiv  Detail & Related papers  (2026-04-15T11:17:54Z)
• See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation [59.07792608884117]
  We introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework.<n>SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure.<n>SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA.
  arXiv  Detail & Related papers  (2026-03-10T07:22:51Z)
• PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation [27.791908160098625]
  PALM builds policy learning around interaction-centric affordance reasoning and subtask progress cues.<n>Palm consistently outperforms baselines in simulation and real-world experiments.
  arXiv  Detail & Related papers  (2026-01-11T21:00:58Z)
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning [61.38273866396522]
  Vision-Language-Action models have emerged as a powerful paradigm for general-purpose robot learning.<n>Current methods remain unsuitable for challenging simulated- or physical-world deployments.<n>We introduce a Test-Time Reinforcement Learning for VLAs framework that enables on-the-fly policy adaptation during inference.
  arXiv  Detail & Related papers  (2026-01-11T01:51:30Z)
• Learning Affordances at Inference-Time for Vision-Language-Action Models [50.93181349331096]
  In robotics, Vision-Language-Action models (VLAs) offer a promising path towards solving complex control tasks.<n>We introduce Learning from Inference-Time Execution (LITEN), which connects a VLA low-level policy to a high-level VLM that conditions on past experiences.<n>Our approach iterates between a reasoning phase that generates and executes plans for the low-level VLA, and an assessment phase that reflects on the resulting execution.
  arXiv  Detail & Related papers  (2025-10-22T16:43:29Z)
• PARL-MT: Learning to Call Functions in Multi-Turn Conversation with Progress Awareness [57.020401590532686]
  We introduce PARL-MT, a framework that explicitly incorporates progress awareness into LLM training for multi-turn function calling.<n>We show that PARL-MT significantly outperforms existing methods, highlighting the effectiveness of progress awareness in enabling robust and efficient multi-turn function calling.
  arXiv  Detail & Related papers  (2025-09-27T09:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.