RAGEN-2: Reasoning Collapse in Agentic RL

• URL: http://arxiv.org/abs/2604.06268v1
• Date: Tue, 07 Apr 2026 04:29:41 GMT
• Title: RAGEN-2: Reasoning Collapse in Agentic RL
• Abstract summary: In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic.<n>We decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI)<n>Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality.
• Score: 123.40730283427136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: RL training of multi-turn LLM agents is inherently unstable, and reasoning quality directly determines task performance. Entropy is widely used to track reasoning stability. However, entropy only measures diversity within the same input, and cannot tell whether reasoning actually responds to different inputs. In RAGEN-2, we find that even with stable entropy, models can rely on fixed templates that look diverse but are input-agnostic. We call this template collapse, a failure mode invisible to entropy and all existing metrics. To diagnose this failure, we decompose reasoning quality into within-input diversity (Entropy) and cross-input distinguishability (Mutual Information, MI), and introduce a family of mutual information proxies for online diagnosis. Across diverse tasks, mutual information correlates with final performance much more strongly than entropy, making it a more reliable proxy for reasoning quality. We further explain template collapse with a signal-to-noise ratio (SNR) mechanism. Low reward variance weakens task gradients, letting regularization terms dominate and erase cross-input reasoning differences. To address this, we propose SNR-Aware Filtering to select high-signal prompts per iteration using reward variance as a lightweight proxy. Across planning, math reasoning, web navigation, and code execution, the method consistently improves both input dependence and task performance.

Related papers

• Deconstructing Off-Policy Ratios: Entropy-Scaled Trust Regions for Asynchronous Reinforcement Learning [6.667218294299622]
  Asynchronous reinforcement learning (RL) accelerates large language model (LLM) post-training by overlapping rollout generation with policy optimization.<n>Existing methods typically retain or discard tokens based solely on the magnitude of their importance ratios.<n>We propose the Entropy-Scaled Trust Region (ESTR), which scales each token's off-policy deviation by its local entropy.
  arXiv  Detail & Related papers  (2026-07-24T10:55:42Z)
• Demystifying Variance in Circuit Discovery of LLMs [51.97489679448601]
  We develop a new circuit discovery method that substantially lessens resampling variance.<n>We show that rephrasing variance arises because prompts with different templates tend to activate different circuits in the model.<n>We argue that sparsity, which has been claimed to form more compact and interpretable task circuits, fails to solve this problem.
  arXiv  Detail & Related papers  (2026-06-15T16:25:07Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• M2A: Synergizing Mathematical and Agentic Reasoning in Large Language Models [9.712139528293946]
  We propose M2A, a novel paradigm that synergizes mathematical and agentic reasoning via model merging.<n>M2A operates directly in parameter space, identifies the feature subspace critical for agent behavior, and merges the mathematical reasoning task vector only along its null space.<n>Unlike SFT or RL, M2A requires no additional gradient-update and exposes the coefficient merging as a simple knob for controlling reasoning length.
  arXiv  Detail & Related papers  (2026-05-11T02:05:30Z)
• Agentic Test-Time Scaling for WebAgents [65.5178428849495]
  We present Confidence-Aware Test-Time Scaling (CATTS), which uses vote-derived uncertainty to allocate compute only when decisions are genuinely contentious.<n>CATTS improves performance on WebArena-Lite and GoBrowse by up to 9.1% over React while using up to 2.3x fewer tokens than uniform scaling.
  arXiv  Detail & Related papers  (2026-02-12T18:58:30Z)
• Failure Modes in Multi-Hop QA: The Weakest Link Law and the Recognition Bottleneck [38.430205063059724]
  We introduce Multi-Focus Attention Instruction (MFAI) to disentangle mechanisms by explicitly steering attention towards selected positions.<n>We demonstrate that "thinking" models that utilize System-2 reasoning, effectively locate and integrate the required information.
  arXiv  Detail & Related papers  (2026-01-18T17:16:04Z)
• Differential Smoothing Mitigates Sharpening and Improves LLM Reasoning [23.202550214006237]
  It is widely recognized that reinforcement learning (RL) fine-tuning of large language models often leads to textitdiversity collapse, where outputs lack variety.<n>We introduce a principled method -- textitdifferential smoothing -- that provably improves both correctness and diversity, outperforming vanilla RL as well as widely used entropy-based languages.
  arXiv  Detail & Related papers  (2025-11-25T05:28:55Z)
• Abduct, Act, Predict: Scaffolding Causal Inference for Automated Failure Attribution in Multi-Agent Systems [20.846301581161978]
  Failure attribution in multi-agent systems is a critical yet unsolved challenge.<n>Current methods treat this as a pattern recognition task over long conversation logs.<n>A2P Scaffolding transforms failure attribution from pattern recognition into a structured causal inference task.
  arXiv  Detail & Related papers  (2025-09-12T16:51:15Z)
• The Choice of Divergence: A Neglected Key to Mitigating Diversity Collapse in Reinforcement Learning with Verifiable Reward [57.56453588632619]
  A central paradox in fine-tuning Large Language Models (LLMs) with Reinforcement Learning with Verifiable Reward (RLVR) is the frequent degradation of multi-attempt performance.<n>This is often accompanied by catastrophic forgetting, where models lose previously acquired skills.<n>We argue that standard RLVR objectives lack a crucial mechanism for knowledge retention.
  arXiv  Detail & Related papers  (2025-09-09T06:34:32Z)
• Adapt in the Wild: Test-Time Entropy Minimization with Sharpness and Feature Regularization [85.50560211492898]
  Test-time adaptation (TTA) may fail to improve or even harm the model performance when test data have mixed distribution shifts.<n>This is often a key obstacle preventing existing TTA methods from being deployed in the real world.<n>We propose a sharpness-aware and reliable entropy minimization method, called SAR, for stabilizing TTA from two aspects.
  arXiv  Detail & Related papers  (2025-09-05T10:03:00Z)
• Inv-Entropy: A Fully Probabilistic Framework for Uncertainty Quantification in Language Models [12.743668975795144]
  Large language models (LLMs) have transformed natural language processing, but their reliable deployment requires effective uncertainty quantification (UQ)<n>Existing UQ methods are often and lack a probabilistic interpretation.<n>We propose a fully probabilistic framework based on an inverse model, which quantifies uncertainty by evaluating the diversity of the input space conditioned on a given output through systematic perturbations.
  arXiv  Detail & Related papers  (2025-06-11T13:02:17Z)
• Detecting Rewards Deterioration in Episodic Reinforcement Learning [63.49923393311052]
  In many RL applications, once training ends, it is vital to detect any deterioration in the agent performance as soon as possible. We consider an episodic framework, where the rewards within each episode are not independent, nor identically-distributed, nor Markov. We define the mean-shift in a way corresponding to deterioration of a temporal signal (such as the rewards), and derive a test for this problem with optimal statistical power.
  arXiv  Detail & Related papers  (2020-10-22T12:45:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.