SE-Search: Self-Evolving Search Agent via Memory and Dense Reward

• URL: http://arxiv.org/abs/2603.03293v1
• Date: Fri, 06 Feb 2026 09:14:07 GMT
• Title: SE-Search: Self-Evolving Search Agent via Memory and Dense Reward
• Abstract summary: Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge.<n>Existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals.<n>We propose textbfSelf-textbfEvolving textbfSearch, a Self-Evolving Search agent that improves online search behavior through three components.
• Score: 87.79131676521656
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge. Recent search agents further cast RAG as an autonomous, multi-turn information-seeking process. However, existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals. We propose \textbf{S}elf-\textbf{E}volving \textbf{Search}, a Self-Evolving Search agent that improves online search behavior through three components, memory purification, atomic query training, and dense rewards. SE-Search follows a \textit{Think-Search-Memorize} strategy that retains salient evidence while filtering irrelevant content. Atomic query training promotes shorter and more diverse queries, improving evidence acquisition. Dense rewards provide fine-grained feedback that speeds training. Experiments on single-hop and multi-hop question answering benchmarks show that \texttt{SE-Search-3B} outperforms strong baselines, yielding a $10.8$ point absolute improvement and a $33.8\%$ relative gain over Search-R1.\footnote{We will make the code and model weights publicly available upon acceptance.}

Related papers

• SearchMaster: Grounded and Regulated Self-Play for Search Agents [57.537666863340284]
  Training search agents requires high-quality search data.<n> Existing pipelines often depend on human-written tasks, expert demonstrations, or stronger teacher models.<n>We present SearchMaster, a self-play framework that trains a single LLM from search tasks it generates, solves, and verifies in a local search environment.
  arXiv  Detail & Related papers  (2026-08-03T07:29:11Z)
• RING: Retrieval-Internalized Generation for Continual Large-Scale Knowledge Injection [88.56816599695391]
  Retrieval-augmented generation (RAG) improves factuality but adds latency and engineering overhead at serving time.<n>We propose RING (Retrieval-Internalized Generation), a holistic paradigm spanning both architecture and training.
  arXiv  Detail & Related papers  (2026-08-03T03:00:43Z)
• Superintelligent Retrieval Agent: The Next Frontier of Information Retrieval [25.731213365755234]
  We introduce textitSuperIntelligent Retrieval Agent (SIRA)<n>SIRA can compress multi-round exploratory search into a single corpus-discriminative retrieval action.<n>It can exceed substantially more expensive multi-round search while remaining interpretable, training-free, and efficient.
  arXiv  Detail & Related papers  (2026-05-07T17:54:29Z)
• PRAISE: Prefix-Based Rollout Reuse in Agentic Search Training [28.912613644535668]
  We present PRAISE, a framework for improving data efficiency and credit assignment in agentic search training.<n>Our method uses a single shared model for both search policy learning and prefix answer evaluation.<n>Experiments on multi-hop QA benchmarks show that PRAISE consistently improves performance over strong baselines.
  arXiv  Detail & Related papers  (2026-04-04T10:23:46Z)
• Meta-Reinforcement Learning with Self-Reflection for Agentic Search [101.39929522022514]
  This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection.<n>Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes.
  arXiv  Detail & Related papers  (2026-03-11T21:40:26Z)
• Agentic Search in the Wild: Intents and Trajectory Dynamics from 14M+ Real Search Requests [28.788329427390455]
  This paper presents a large-scale log analysis of agentic search based on 14.44M search requests (3.97M sessions) collected from DeepResearch.<n>Our analyses reveal distinctive behavioral patterns. First, over 90% of multi-turn sessions contain at most ten steps, and 89% of inter-step intervals fall under one minute.<n>Findings suggest that agentic search may benefit from repetition-aware early stopping, intent-adaptive retrieval budgets, and explicit cross-step context tracking.
  arXiv  Detail & Related papers  (2026-01-24T22:42:43Z)
• Agentic-R: Learning to Retrieve for Agentic Search [82.43426728778418]
  We propose a novel retriever training framework tailored for agentic search.<n>Unlike retrievers designed for single-turn retrieval-augmented generation (RAG), we propose to use both local query-passage relevance and global answer correctness.<n>Our retriever, termed ours, consistently outperforms strong baselines across different search agents.
  arXiv  Detail & Related papers  (2026-01-17T02:59:54Z)
• LEAPS: An LLM-Empowered Adaptive Plugin for Taobao AI Search [17.074638179635613]
  We propose LEAPS (LLM-Empowered Adaptive for Taobao AI Search), which seamlessly upgrades traditional search systems via a "Broaden-and-Empower" paradigm.<n>It employs a novel three-stage training strategy--inverse data augmentation, posterior-knowledge supervised fine-tuning, and diversity-aware reinforcement learning--to generate adaptive and complementary query combinations.<n>Fully deployed on Taobao AI Search since August 2025, LEAPS currently serves hundreds of millions of users monthly.
  arXiv  Detail & Related papers  (2026-01-09T03:41:27Z)
• MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learning [73.27233666920618]
  We propose MemSearcher, an agent workflow that iteratively maintains a compact memory and combines the current turn with it.<n>At each turn, MemSearcher fuses the user's question with the memory to generate reasoning traces, perform search actions, and update memory to retain only information essential for solving the task.<n>We introduce multi-context GRPO, an end-to-end RL framework that jointly optimize reasoning, search strategies, and memory management of MemSearcher Agents.
  arXiv  Detail & Related papers  (2025-11-04T18:27:39Z)
• Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learning [23.104182075898297]
  Reasoning-augmented search agents, such as Search-R1, are trained to reason, search, and generate the final answer iteratively.<n>We train an LLM-based search agent with the native capability of query expansion through reinforcement learning.<n>With the assistance of the squeezer model, we discover that even a small-scale 3B LLM can demonstrate a strong capability of query expansion.
  arXiv  Detail & Related papers  (2025-10-11T04:23:30Z)
• Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents [19.31471304268234]
  We introduce DeSA (Decoupling Search-and-Answering), a simple two-stage training framework that explicitly separates search optimization from answer generation.<n>Across seven QA benchmarks, DeSA-trained agents consistently improve search behaviors, delivering substantially higher search recall and answer accuracy than outcome-only baselines.
  arXiv  Detail & Related papers  (2025-10-06T11:09:45Z)
• Fathom-DeepResearch: Unlocking Long Horizon Information Retrieval and Synthesis for SLMs [7.3517692707289415]
  We introduce Fathom-DeepResearch, an agentic system composed of two specialized models.<n>The first is Fathom-Search-4B, a DeepSearch model optimized for evidence-based investigation through live web search and targeted webpage querying.<n>The second is Fathom- Synthesizer-4B, trained from Qwen3-4B, which converts multi-turn DeepSearch traces into structured, citation-dense DeepResearch Reports.
  arXiv  Detail & Related papers  (2025-09-28T22:58:11Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning [74.65632662894086]
  We propose ReSearch, a framework that trains LLMs to Reason with Search via reinforcement learning.<n>Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking.<n>Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction.
  arXiv  Detail & Related papers  (2025-03-25T09:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.