Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents

• URL: http://arxiv.org/abs/2608.06861v1
• Date: Fri, 07 Aug 2026 06:38:12 GMT
• Title: Gated-BEPO: Confidence-Gated Bellman Credit Assignment for Large Language Model Agents
• Abstract summary: Training large language model agents in long-horizon environments requires assigning credit from sparse terminal outcomes to individual actions.<n>We propose Gated-BEPO, which derives step-level credit from empirical rollout graphs.
• Score: 20.885871857379897
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large language model agents in long-horizon environments requires assigning credit from sparse terminal outcomes to individual actions. Existing critic-free methods propagate trajectory-level rewards uniformly across steps, while recent approaches construct step-level groups by matching repeated states and compare actions within each group. The former cannot distinguish useful actions in failed trajectories from ineffective actions in successful ones. The latter rely on step credit derived directly from individual trajectory outcomes and fixed-weight fusion with episode-level credit. We propose Gated-BEPO, which derives step-level credit from empirical rollout graphs. For each rollout group, Gated-BEPO constructs an empirical graph and estimates node values through a mean-backup Bellman fixed point that reflects the empirical action distribution of the current policy. We then accumulate these temporal-difference residuals along each sampled trajectory using generalized advantage estimation, yielding step-level Bellman advantages that capture both immediate and downstream effects. To adaptively fuse episode- and step-level credit, a confidence gate incorporates Bellman credit only at states with multiple observed successors and otherwise uses episode-level credit. Experiments on WebShop, ALFWorld, and visual Sokoban show consistent improvements across language and vision-language models, while diagnostic ablations support the effectiveness of Bellman fixed-point value estimation and show that step-level credit should be incorporated selectively rather than uniformly into the final advantage.

Related papers

• PGPO: Potential-Guided Policy Optimization for Multi-Turn Agentic Tasks [41.82007055765541]
  We propose Potential-Guided Policy Optimization for multi-turn agentic tasks.<n>It estimates empirical state potentials from anchor-state-group return statistics within each rollout group.<n>It then derives action advantages from potential differences between adjacent states, enabling cross-trajectory credit propagation.
  arXiv  Detail & Related papers  (2026-09-02T07:44:16Z)
• MileGPO: Milestone Inference with Local Evidence for Graph-Based Policy Optimization of Long-Horizon LLM Agents [26.262385802144323]
  We propose MileGPO, which derives process-level credit from grouped on-policy rollouts through three designs.<n>MileGPO requires neither auxiliary models nor additional environment interaction.
  arXiv  Detail & Related papers  (2026-08-20T08:58:27Z)
• PACE: Phase-Progress-Aware Credit for Long-Horizon Embodied Manipulation [17.66354304197193]
  Policy improvement requires step-level credit signals to distinguish behaviors that advance the task from those that stall or regress.<n>We present PACE, a credit-assignment framework for post-training on long-horizon manipulation.
  arXiv  Detail & Related papers  (2026-08-15T04:25:16Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• GradCuit: Credit-Assigned Gradient Flow Enables Robust and Interpretable Test-Time Latent Reasoning [61.67411833252235]
  We introduce GradCuit, which inserts optimizable latent states at a selected Transformer layer between the hidden representations of the prompt and the generated continuation.<n>Across five instruction-tuned backbones, three reasoning benchmarks, and two answer formats, GradCuit achieves an average accuracy of 64.5%.<n>For interpretability, token-level gradient attribution reveals that latent influence concentrates on reasoning-connector tokens, while layer analysis identifies early-to-middle Transformer layers as the most effective optimization space.
  arXiv  Detail & Related papers  (2026-08-03T17:55:24Z)
• Semantic Consistency Policy Optimization for Reinforcement Learning of LLM Agents [45.62262776712415]
  Group-based reinforcement learning effectively post-trains agents for long-horizon, sparse-reward tasks by deriving step-level credit from trajectory outcomes.<n>This inconsistency sends conflicting gradients to similar actions and wastes the partially-correct progress inside failed rollouts.<n>We propose Semantic Consistency Policy Optimization ( SCPO), a value-free reward-shaping method that mitigates this inconsistency.<n> SCPO scores each failed step against a successful sibling and adds positive step-level credit for new progress along that sibling.
  arXiv  Detail & Related papers  (2026-06-24T14:02:13Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment [35.978805768172656]
  Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning.<n>We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards.
  arXiv  Detail & Related papers  (2026-06-08T11:20:58Z)
• Potential-Guided Flow Matching for Vision-Language-Action Policy Improvement [27.695600755960736]
  We introduce ForesightFlow, a self-guided flow-matching policy.<n>It augments each generated action chunk with a learned success-potential trajectory.<n>It scores candidate actions, enabling best-of-$K$ inference without an external critic.
  arXiv  Detail & Related papers  (2026-06-03T14:49:35Z)
• VeriGate: Verifier-Gated Step-Level Supervision for GRPO [51.26100506256885]
  Group Relative Policy Optimization is an effective recipe for training reasoning models with verifier-based outcome rewards.<n>We present VeriGate, a verifier-gated extension of GRPO that addresses these limitations with three design choices.<n>We show that VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively.
  arXiv  Detail & Related papers  (2026-05-28T18:20:32Z)
• StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning [6.365332042924078]
  We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution.<n>StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts.<n>Results reveal a consistent two-knob law: smaller _clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength _mix remains task-dependent.
  arXiv  Detail & Related papers  (2026-05-26T15:07:03Z)
• GAGPO: Generalized Advantage Grouped Policy Optimization [13.085463559931156]
  We propose a critic-free reinforcement learning method for precise, step-aligned temporal credit assignment.<n>Experiments on ALFWorld and WebShop show that GAGPO outperforms strong reinforcement learning baselines.
  arXiv  Detail & Related papers  (2026-05-13T09:10:03Z)
• Learning to Credit the Right Steps: Objective-aware Process Optimization for Visual Generation [57.61006679699323]
  Reinforcement learning has emerged as an effective framework for post-training visual generative models with human preference signals.<n>In modern visual generation, multiple reward models are often used to capture heterogeneous objectives, such as visual quality, motion consistency, and text alignment.<n>We propose Objective-aware Trajectory Credit Assignment (OTCA), a structured framework for fine-grained GRPO training.
  arXiv  Detail & Related papers  (2026-04-21T08:37:13Z)
• Prune-Then-Plan: Step-Level Calibration for Stable Frontier Exploration in Embodied Question Answering [52.69447404069251]
  Large vision-language models (VLMs) have improved embodied question answering (EQA) agents by providing strong semantic priors for open-vocabulary reasoning.<n>We propose Prune-Then-Plan, a framework that stabilizes exploration through step-level calibration.
  arXiv  Detail & Related papers  (2025-11-24T22:50:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.