Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation

• URL: http://arxiv.org/abs/2608.04788v1
• Date: Wed, 05 Aug 2026 12:52:14 GMT
• Title: Agentic Reinforcement Learning with Observation-Calibrated Self-Distillation
• Abstract summary: On-Policy Self-Distillation (OPSD) re-scores generated tokens under a privileged replay view to obtain dense, token-level supervision.<n>The resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold.<n>We propose Observation-Calibrated Self-Distillation (OCSD) to resolve this confounding.
• Score: 18.20369971970953
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language model agents are commonly trained through reinforcement learning with sparse trajectory-level rewards, which offer limited guidance on how strongly individual tokens should be updated. On-Policy Self-Distillation (OPSD) addresses this by re-scoring generated tokens under a privileged replay view to obtain dense, token-level supervision. However, we identify a confounding issue: the resulting support may reflect both the privileged information contained in the replay view and score shifts induced by the replay scaffold, making it difficult to attribute the support specifically to that information. This issue is especially pronounced when future environment observations serve as privileged information, since replaying them requires reconstructing an extended scaffold that itself perturbs token scores. To resolve this confounding, we propose Observation-Calibrated Self-Distillation (OCSD), which contrasts two structurally matched replay views, Full and Observation-Ablated, differing only in whether the actual future observation is present, to derive an observation residual that discounts score changes shared by the replay scaffold. OCSD then applies this residual to modulate token-level GRPO updates at high-uncertainty steps, while preserving the trajectory-level update direction. Experiments on ALFWorld, WebShop, and Search-QA across three Qwen3 model scales show that OCSD consistently outperforms strong baselines. Diagnostic analyses further confirm that the calibrated residual aligns better with local environment feedback. Our code is publicly available at https://github.com/yiy1x/OCSD.

Related papers

• Inside VLM Chart Reading: Tracing Value Reading from Vertical Bar Charts Across Space and Depth [50.62320148925983]
  Vision--language models (VLMs) can answer chart questions accurately, but output accuracy does not show how they combine the evidence needed to recover an exact value.<n>We study vertical-bar value reading with controlled counterfactual activation patching in Qwen2.5VL-7B-Instruct and InternVL3.5-8B.
  arXiv  Detail & Related papers  (2026-09-12T06:28:11Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• Self-Review Reinforcement Learning (SRRL) with Cross-Episode Memory and Policy Distillation [0.0]
  Reinforcement Learning is commonly used to train large language models using environmental feedback.<n>We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode.<n>When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt.
  arXiv  Detail & Related papers  (2026-07-06T18:24:16Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• When Self-Belief Misleads: Active Label Acquisition for Reinforcement Learning with Verifiable Rewards [49.25249414962884]
  We propose Reinforcement Learning with Active Verifiable Rewards (RLAVR)<n>RLAVR actively acquires ground-truth labels for a small set of selected samples and integrates them with pseudo-labels.<n>Building on this, we introduce Correction-Aware Reliability Estimation for RLAVR (CARE)
  arXiv  Detail & Related papers  (2026-05-25T13:55:12Z)
• Clipping Bottleneck: Stabilizing RLVR via Stochastic Recovery of Near-Boundary Signals [83.0127582612634]
  Near-boundary Rescue (NSR) is a minimal, plug-and-play modification that retains out-of-bound tokens to recover lost signals.<n>NSR substantially improves training stability and delivers consistent gains over strong baselines such as DAPO and GSPO.
  arXiv  Detail & Related papers  (2026-05-21T16:45:31Z)
• Rewarding Beliefs, Not Actions: Consistency-Guided Credit Assignment for Long-Horizon Agents [5.917866758929418]
  Reinforcement learning from verifiable rewards (RLVR) is a promising paradigm for improving large language model (LLM) agents on long-horizon interactive tasks.<n>We propose ReBel (Reward Belief), a process-level reinforcement learning algorithm that explicitly models structured belief states.<n>We evaluate ReBel on challenging long-horizon benchmarks, including ALFWorld and WebShop.
  arXiv  Detail & Related papers  (2026-05-19T16:19:29Z)
• AMR-SD: Asymmetric Meta-Reflective Self-Distillation for Token-Level Credit Assignment [39.63424981516754]
  Asymmetric Meta-Reflective Self-Distillation (AMR-SD)<n>We introduce Causal Information Gain (CIG) with an asymmetric, ReLU-gated threshold to translate these reflections into sparse, highly precise token-level advantage modulations.<n> Experiments across scientific, mathematical, and tool-use benchmarks demonstrate that AMR-SD significantly outperforms existing baselines.
  arXiv  Detail & Related papers  (2026-05-18T15:14:34Z)
• Learning Uncertainty from Sequential Internal Dispersion in Large Language Models [52.29267172760918]
  Uncertainty estimation is a promising approach to detect hallucinations in large language models.<n>Recent approaches commonly depend on model internal states to estimate uncertainty.<n>We present Sequential Internal Variance Representation (SIVR), a supervised hallucination detection framework.
  arXiv  Detail & Related papers  (2026-04-17T06:31:29Z)
• Dual-Stage Invariant Continual Learning under Extreme Visual Sparsity [8.16821029459195]
  We show that background-driven gradients destabilize the feature backbone during sequential domain shifts.<n>We propose a dual-stage invariant continual learning framework via joint distillation.<n> Experiments on a high-resolution space-based RSO detection dataset show consistent improvement over established continual object detection methods.
  arXiv  Detail & Related papers  (2026-03-27T09:03:49Z)
• CORE: Context-Robust Remasking for Diffusion Language Models [51.59514489363897]
  We propose Context-Robust Remasking (CORE), a training-free framework for inference-time revision.<n>Rather than trusting static token probabilities, CORE identifies context-brittle tokens by probing their sensitivity to targeted masked-context perturbations.<n>On LLaDA-8B-Base, CORE delivers consistent improvements across reasoning and code benchmarks, outperforming compute-matched baselines and improving MBPP by up to 9.2 percentage points.
  arXiv  Detail & Related papers  (2026-02-04T00:12:30Z)
• Anticipating the Unseen Discrepancy for Vision and Language Navigation [63.399180481818405]
  Vision-Language Navigation requires the agent to follow natural language instructions to reach a specific target. The large discrepancy between seen and unseen environments makes it challenging for the agent to generalize well. We propose Unseen Discrepancy Anticipating Vision and Language Navigation (DAVIS) that learns to generalize to unseen environments via encouraging test-time visual consistency.
  arXiv  Detail & Related papers  (2022-09-10T19:04:40Z)
• Robust Disentanglement of a Few Factors at a Time [5.156484100374058]
  We introduce population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) We then use Unsupervised Disentanglement Ranking (UDR) as an unsupervised to score models in our PBT-VAE training and show how models trained this way tend to consistently disentangle only a subset of the generative factors. We show striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics.
  arXiv  Detail & Related papers  (2020-10-26T12:34:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.