On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents

• URL: http://arxiv.org/abs/2603.12109v1
• Date: Thu, 12 Mar 2026 16:14:14 GMT
• Title: On Information Self-Locking in Reinforcement Learning for Active Reasoning of LLM agents
• Abstract summary: In active reasoning, large language model (LLM) agents often suffer from information self-locking.<n>We decompose active reasoning into two core capabilities: Action Selection (AS) and Belief Tracking (BT)<n>We show that deficient AS and BT capabilities will limit the information exploration during RL training.<n>We propose a simple yet effective approach that reallocates the learning signal by injecting easy- to-obtain directional critiques.
• Score: 33.32413617717957
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) with outcome-based rewards has achieved significant success in training large language model (LLM) agents for complex reasoning tasks. However, in active reasoning where agents need to strategically ask questions to acquire task-relevant information, we find that LLM agents trained with RL often suffer from information self-locking: the agent ceases to ask informative questions and struggles to internalize already-obtained information. To understand the phenomenon, we decompose active reasoning into two core capabilities: Action Selection (AS), which determines the observation stream through queries, and Belief Tracking (BT), which updates the agent's belief based on collected evidence. We show that deficient AS and BT capabilities will limit the information exploration during RL training. Furthermore, insufficient exploration in turn hinders the improvement of AS and BT, creating a feedback loop that locks the agent in a low-information regime. To resolve the issue, we propose a simple yet effective approach that reallocates the learning signal by injecting easy- to-obtain directional critiques to help the agent escape self-locking. Extensive experiments with 7 datasets show that our approach significantly mitigates the information self-locking, bringing up to 60% improvements.

Related papers

• AgentV-RL: Scaling Reward Modeling with Agentic Verifier [63.55502685076245]
  Verifiers have been demonstrated to enhance LLM reasoning via test-time scaling (TTS)<n>We propose Agentic Verifier, a framework that transforms reward modeling into a multi-turn, tool-augmented deliberative process.<n>We show that Agentic Verifier yields consistent performance gains under both parallel and sequential TTS.
  arXiv  Detail & Related papers  (2026-04-17T12:27:36Z)
• Secure Forgetting: A Framework for Privacy-Driven Unlearning in Large Language Model (LLM)-Based Agents [20.770088583807148]
  This paper initiates research on unlearning in large language model (LLM)-based agents.<n>We propose a novel and comprehensive framework that categorizes unlearning scenarios into three contexts.<n>Within this framework, we introduce a natural language-based unlearning method that trains a conversion model to transform high-level unlearning requests into actionable unlearning prompts.
  arXiv  Detail & Related papers  (2026-04-01T03:17:35Z)
• Learning to Retrieve from Agent Trajectories [72.8923565916533]
  We argue that retrieval models for agentic search should be trained directly from agent interaction data.<n>We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions.<n>Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
  arXiv  Detail & Related papers  (2026-03-30T17:59:02Z)
• Thinking Makes LLM Agents Introverted: How Mandatory Thinking Can Backfire in User-Engaged Agents [23.785816075149484]
  Eliciting reasoning has emerged as a powerful technique for improving the performance of large language models (LLMs) on complex tasks by inducing thinking.<n>We conduct a comprehensive study on the effect of explicit thinking in user-engaged LLM agents.<n>We find that mandatory thinking often backfires on agents in user-engaged settings, causing anomalous performance degradation.
  arXiv  Detail & Related papers  (2026-02-08T03:23:22Z)
• The Path of Self-Evolving Large Language Models: Achieving Data-Efficient Learning via Intrinsic Feedback [51.144727949988436]
  Reinforcement learning (RL) has demonstrated potential to enhance the reasoning capabilities of large language models (LLMs)<n>In this work, we explore improving LLMs through RL with minimal data.<n>To minimize data dependency, we introduce two novel mechanisms grounded in self-awareness.
  arXiv  Detail & Related papers  (2025-10-03T06:32:10Z)
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards [18.92867715736209]
  We propose ReSeek, a novel self-correcting framework for training search agents.<n>Our framework introduces a self-correction mechanism that empowers the agent to dynamically identify and recover from erroneous search paths.<n>To mitigate the risk of data contamination in existing datasets, we introduce FictionalHot.
  arXiv  Detail & Related papers  (2025-10-01T06:44:28Z)
• An Analysis of Decoding Methods for LLM-based Agents for Faithful Multi-Hop Question Answering [44.41915467956464]
  Large Language Models (LLMs) frequently produce factually inaccurate outputs.<n>This phenomenon limits their accuracy in knowledge-intensive NLP tasks.<n>Recent research has explored training-free decoding strategies to improve the faithfulness of model generations.
  arXiv  Detail & Related papers  (2025-03-30T12:18:21Z)
• LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage [78.33839735526769]
  LeakAgent is a novel black-box red-teaming framework for privacy leakage.<n>Our framework trains an open-source LLM through reinforcement learning as the attack agent to generate adversarial prompts.<n>We show that LeakAgent significantly outperforms existing rule-based approaches in training data extraction and automated methods in system prompt leakage.
  arXiv  Detail & Related papers  (2024-12-07T20:09:01Z)
• ThinkNote: Enhancing Knowledge Integration and Utilization of Large Language Models via Constructivist Cognition Modeling [55.21641515545307]
  Large Language Models (LLMs) have demonstrated strong performance across a wide range of NLP tasks.<n>They often exhibit suboptimal behaviors and inconsistencies when exposed to unfamiliar external information.<n>We propose ThinkNote, a novel framework that enhances the external knowledge utilization of LLMs.
  arXiv  Detail & Related papers  (2024-02-21T06:04:53Z)
• A Closer Look at the Limitations of Instruction Tuning [52.587607091917214]
  We show that Instruction Tuning (IT) fails to enhance knowledge or skills in large language models (LLMs) We also show that popular methods to improve IT do not lead to performance improvements over a simple LoRA fine-tuned model. Our findings reveal that responses generated solely from pre-trained knowledge consistently outperform responses by models that learn any form of new knowledge from IT on open-source datasets.
  arXiv  Detail & Related papers  (2024-02-03T04:45:25Z)
• PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-training [94.87393610927812]
  We present an off-policy, interactive reinforcement learning algorithm that capitalizes on the strengths of both feedback and off-policy learning. We demonstrate that our approach is capable of learning tasks of higher complexity than previously considered by human-in-the-loop methods.
  arXiv  Detail & Related papers  (2021-06-09T14:10:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.