LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent

• URL: http://arxiv.org/abs/2604.17931v2
• Date: Wed, 22 Apr 2026 09:13:22 GMT
• Title: LiteResearcher: A Scalable Agentic RL Training Framework for Deep Research Agent
• Abstract summary: LiteResearcher is a training framework that makes Agentic RL scalable.<n>By constructing a lite virtual world that mirrors real-world search dynamics, we enable a tiny search agent to outperform large-scale open-source and commercial models.
• Score: 20.58731853877092
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement Learning (RL) has emerged as a powerful training paradigm for LLM-based agents. However, scaling agentic RL for deep research remains constrained by two coupled challenges: hand-crafted synthetic data fails to elicit genuine real-world search capabilities, and real-world search dependency during RL training introduces instability and prohibitive cost, which limits the scalability of Agentic RL. LiteResearcher is a training framework that makes Agentic RL scalable: by constructing a lite virtual world that mirrors real-world search dynamics, we enable a continuously improving training recipe that empowers a tiny search agent to outperform large-scale open-source and commercial models (e.g., Tongyi DeepResearch and Claude-4.5 Sonnet). Specifically, on common benchmarks such as GAIA and Xbench, our LiteResearcher-4B achieves open-source state-of-the-art results of 71.3% and 78.0% respectively, demonstrating that scalable RL training is a key enabler for Deep Research Agents.

Related papers

• OffSeeker: Online Reinforcement Learning Is Not All You Need for Deep Research Agents [68.79728889531806]
  State-of-the-art performance typically relies on online reinforcement learning (RL)<n>We introduce a fully open-source suite designed for effective offline training.<n>Our core contributions include DeepForge, a ready-to-use task synthesis framework that generates large-scale research queries without heavy preprocessing.
  arXiv  Detail & Related papers  (2026-01-26T13:13:59Z)
• Search Self-play: Pushing the Frontier of Agent Capability without Supervision [14.889394507446477]
  Self-play training for deep search agents is proposed in this paper.<n>In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation.<n>SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision.
  arXiv  Detail & Related papers  (2025-10-21T17:19:35Z)
• Demystifying Reinforcement Learning in Agentic Reasoning [90.3737088727791]
  We conduct a comprehensive and systematic investigation to demystify reinforcement learning in agentic reasoning.<n>We highlight our key insights: (i) replacing stitched synthetic trajectories with real end-to-end tool-use trajectories yields a far stronger SFT.<n> Exploration-friendly techniques are crucial for agentic RL, such as clip higher, overlong reward shaping, and maintaining adequate policy entropy could improve the training efficiency.
  arXiv  Detail & Related papers  (2025-10-13T17:57:15Z)
• A Survey of Reinforcement Learning for Large Reasoning Models [98.58081012669369]
  Review of recent advances in Reinforcement Learning for reasoning with Large Language Models.<n>Further scaling of RL for LRMs now faces challenges not only in computational resources but also in algorithm design, training data, and infrastructure.
  arXiv  Detail & Related papers  (2025-09-10T17:59:43Z)
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning [129.44038804430542]
  We introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL.<n>We propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization.<n>Our agents match or surpass commercial models on 27 tasks across diverse environments.
  arXiv  Detail & Related papers  (2025-09-10T16:46:11Z)
• SSRL: Self-Search Reinforcement Learning [37.21683758480754]
  We quantify the intrinsic search capability of large language models (LLMs)<n>We introduce Self-Search RL (SSRL), which enhances Self-Search capability through format-based and rule-based rewards.<n> Empirical evaluations demonstrate that SSRL-trained policy models provide a cost-effective and stable environment for search-driven RL training.
  arXiv  Detail & Related papers  (2025-08-14T17:46:01Z)
• An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents [34.25887147052966]
  Reinforcement learning (RL) has demonstrated strong potential in training large language models (LLMs) capable of complex reasoning for real-world problem solving.<n>More recently, RL has been leveraged to create sophisticated LLM-based search agents that adeptly combine reasoning with search engine use.<n>Key factors -- such as (1) reward formulation, (2) the choice and characteristics of the underlying LLM, and (3) the role of the search engine in the RL process -- require further investigation.
  arXiv  Detail & Related papers  (2025-05-21T05:09:43Z)
• ZeroSearch: Incentivize the Search Capability of LLMs without Searching [69.55482019211597]
  We introduce ZeroSearch, a framework that incentivizes the capabilities of large language models to use a real search engine with simulated searches during training.<n>Our approach begins with lightweight supervised fine-tuning to transform the LLM into a retrieval module capable of generating both useful and noisy documents.
  arXiv  Detail & Related papers  (2025-05-07T17:30:22Z)
• R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning [87.30285670315334]
  textbfR1-Searcher is a novel two-stage outcome-based RL approach designed to enhance the search capabilities of Large Language Models.<n>Our framework relies exclusively on RL, without requiring process rewards or distillation for a cold start.<n>Our experiments demonstrate that our method significantly outperforms previous strong RAG methods, even when compared to the closed-source GPT-4o-mini.
  arXiv  Detail & Related papers  (2025-03-07T17:14:44Z)
• Auto-Agent-Distiller: Towards Efficient Deep Reinforcement Learning Agents via Neural Architecture Search [15.3602645148428]
  We propose an Auto-Agent-Distiller (A2D) framework to automatically search for the optimal DRL agents for various tasks.<n>We demonstrate that vanilla NAS can easily fail in searching for the optimal agents, due to its resulting high variance in DRL training stability.<n>We then develop a novel distillation mechanism to distill the knowledge from both the teacher agent's actor and critic to stabilize the searching process and improve the searched agents' optimality.
  arXiv  Detail & Related papers  (2020-12-24T04:07:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.