Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy

• URL: http://arxiv.org/abs/2605.14558v1
• Date: Thu, 14 May 2026 08:33:02 GMT
• Title: Resolving Action Bottleneck: Agentic Reinforcement Learning Informed by Token-Level Energy
• Abstract summary: Policy-gradient methods treat each token in a trajectory equally, leading to uniform credit assignment.<n>We show that such uniform credit assignment largely misallocates token-level training signals.<n>We propose an embarrassingly simple token reweighting approach, ActFocus, that downweights gradients on reasoning tokens.
• Score: 64.72195169444738
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic reinforcement learning trains large language models using multi-turn trajectories that interleave long reasoning traces with short environment-facing actions. Common policy-gradient methods, such as PPO and GRPO, treat each token in a trajectory equally, leading to uniform credit assignment. In this paper, we critically demonstrate that such uniform credit assignment largely misallocates token-level training signals. From an energy-based modeling perspective, we show that token-level training signals, quantified by their correlations with reward variance of different rollouts sampled from a given prompt, concentrate sharply on action tokens rather than reasoning tokens, even though action tokens account for only a small fraction of the trajectory. We refer to this phenomenon as the Action Bottleneck. Motivated by this observation, we propose an embarrassingly simple token reweighting approach, ActFocus, that downweights gradients on reasoning tokens, along with an additional energy-based redistribution mechanism that further increases the weights on action tokens with higher uncertainty. Across four environments and different model sizes, ActFocus consistently outperforms PPO and GRPO, yielding final-step gains of up to 65.2 and 63.7 percentage points, respectively, without any additional runtime or memory cost.

Related papers

• Extremely Sparse Supervision Incentivizes Reasoning Ability [13.22207761303279]
  Large language models demonstrate increasingly strong reasoning capabilities through effective post-training.<n> prevailing post-training methods optimize over massive numbers of tokens, implicitly assuming that effective learning must be token-intensive.<n>We discover a counter-intuitive phenomenon: reasoning can be effectively incentivized by an extremely small fraction of generated tokens.
  arXiv  Detail & Related papers  (2026-09-03T23:38:48Z)
• Token-Level Off-Policy Learning for Faithful Generation Under Distribution Shift [26.752668789722765]
  Token-Level Off-Policy Labeling (TOPL) is an off-policy training paradigm that reframes post-training as a token-level correctness prediction task.<n>We show TOPL achieves strong out-of-distribution generalization across 11 datasets against a diverse set of sequence-level and token-level baselines.
  arXiv  Detail & Related papers  (2026-07-20T03:57:22Z)
• ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning [53.17987412112712]
  Reinforcement Learning (RL) has demonstrated significant potential for improving Vision-Language-Action (VLA) models on complex manipulation tasks.<n>In this work, we first investigate the exploration stagnation bottleneck in current VLA-RL frameworks.<n>We introduce RL Exploration Token (ExToken), a simple yet general framework that condition VLA policies on discrete behavioral priors.
  arXiv  Detail & Related papers  (2026-07-14T16:04:41Z)
• How Does Reasoning Flow? Tracing Attention-Induced Information Flow for Targeted RL in LLMs [52.712343092801326]
  FlowTracer is an RL framework that traces answer-targeted reasoning flow on an attention-weighted directed acyclic graph.<n>It scores tokens by flow throughput, revealing high-impact hubs and aggregation checkpoints that mediate long-range dependencies.<n>These derived importances are used to shape token-level rewards, enabling learning signals to focus precisely on the tokens that route information toward correct answers.
  arXiv  Detail & Related papers  (2026-06-09T09:56:51Z)
• PRPO: Perception-Reinforced Policy Optimization via Token-Level Dynamic Advantage Reshaping [5.89473045822308]
  Reinforcement Learning with Verifiable Rewards (RLVR) has become an effective paradigm for improving the reasoning capability of Large Vision-Language Models (LVLMs)<n>Existing RLVR methods rely on trajectory-level outcome rewards, which assign identical learning signals across all generated tokens.<n>We propose Perception-Reinforced Policy Optimization (PRPO), a token-level reinforcement learning framework that explicitly identifies and reinforces pivotal perceptual tokens.
  arXiv  Detail & Related papers  (2026-06-07T16:06:29Z)
• Cornerstones or Stumbling Blocks? Deciphering the Rock Tokens in On-Policy Distillation [4.624042537090342]
  We investigate high-loss tokens, a token type that--as the most direct signal of student-teacher mismatch under On-Policy Distillation's per-token KL objective--should progressively diminish as training converges according to existing studies.<n>We find that these tokens provide negligible functional contribution to the model's actual reasoning performance.
  arXiv  Detail & Related papers  (2026-05-10T01:41:43Z)
• Mimic Intent, Not Just Trajectories [39.77112205526461]
  We argue this stems from mimicking raw trajectories without understanding the underlying intent.<n>We propose explicitly disentangling behavior intent from execution details in end-2-end IL: Mimic Intent, Not just Trajectories.<n> Experiments on several manipulation benchmarks and on a real robot demonstrate state-of-the-art success rates, superior inference efficiency, robust generalization against disturbances, and effective one-shot transfer.
  arXiv  Detail & Related papers  (2026-02-09T12:44:35Z)
• Attention Illuminates LLM Reasoning: The Preplan-and-Anchor Rhythm Enables Fine-Grained Policy Optimization [56.083511902353365]
  Reinforcement learning (RL) typically applies uniform credit across an entire generation of Large language models.<n>This work positions attention as a privileged substrate that renders the internal logic of LLMs as a mechanistic blueprint of reasoning itself.<n>We introduce three novel RL strategies that dynamically perform targeted credit assignment to critical nodes.
  arXiv  Detail & Related papers  (2025-10-15T13:49:51Z)
• ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Training [15.783265191574392]
  We introduce ZeroTuning: a training-free method that improves LLM performance by applying head-specific attention adjustments to the initial token.<n>We show theoretically that adding lightweight biases to this token's attention logits monotonically controls the entropy of the downstream attention distribution.<n>We present two variants: a supervised mode that calibrates on validation examples, and a novel unsupervised mode that directly minimizes the model's output entropy.
  arXiv  Detail & Related papers  (2025-05-16T22:52:24Z)
• Critical Tokens Matter: Token-Level Contrastive Estimation Enhances LLM's Reasoning Capability [53.51560766150442]
  Critical tokens are elements within reasoning trajectories that significantly influence incorrect outcomes.<n>We present a novel framework for identifying these tokens through rollout sampling.<n>We show that identifying and replacing critical tokens significantly improves model accuracy.
  arXiv  Detail & Related papers  (2024-11-29T18:58:22Z)
• Dense Reward for Free in Reinforcement Learning from Human Feedback [64.92448888346125]
  We leverage the fact that the reward model contains more information than just its scalar output. We use these attention weights to redistribute the reward along the whole completion. Empirically, we show that it stabilises training, accelerates the rate of learning, and, in practical cases, may lead to better local optima.
  arXiv  Detail & Related papers  (2024-02-01T17:10:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.