Meta-Reinforcement Learning with Self-Reflection for Agentic Search

• URL: http://arxiv.org/abs/2603.11327v2
• Date: Wed, 18 Mar 2026 07:07:34 GMT
• Title: Meta-Reinforcement Learning with Self-Reflection for Agentic Search
• Abstract summary: This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection.<n>Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes.
• Score: 101.39929522022514
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper introduces MR-Search, an in-context meta reinforcement learning (RL) formulation for agentic search with self-reflection. Instead of optimizing a policy within a single independent episode with sparse rewards, MR-Search trains a policy that conditions on past episodes and adapts its search strategy across episodes. MR-Search learns to learn a search strategy with self-reflection, allowing search agents to improve in-context exploration at test-time. Specifically, MR-Search performs cross-episode exploration by generating explicit self-reflections after each episode and leveraging them as additional context to guide subsequent attempts, thereby promoting more effective exploration during test-time. We further introduce a multi-turn RL algorithm that estimates a dense relative advantage at the turn level, enabling fine-grained credit assignment on each episode. Empirical results across various benchmarks demonstrate the advantages of MR-Search over baselines based RL, showing strong generalization and relative improvements of 9.2% to 19.3% across eight benchmarks. Our code and data are available at https://github.com/tengxiao1/MR-Search.

Related papers

• FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics [24.125726163497742]
  FML-Bench is a benchmark of 18 fundamental ML research tasks across 10 domains.<n>It separates agent strategy from execution infrastructure and defines 12 process-level behavioral metrics.<n>We find greedy search tends to be more effective when opportunities are dense, while tree-search and evolutionary strategies tend to be more effective when opportunities are sparse.
  arXiv  Detail & Related papers  (2026-05-17T10:30:38Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• PRAISE: Prefix-Based Rollout Reuse in Agentic Search Training [28.912613644535668]
  We present PRAISE, a framework for improving data efficiency and credit assignment in agentic search training.<n>Our method uses a single shared model for both search policy learning and prefix answer evaluation.<n>Experiments on multi-hop QA benchmarks show that PRAISE consistently improves performance over strong baselines.
  arXiv  Detail & Related papers  (2026-04-04T10:23:46Z)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization [64.61432234404276]
  emphSearch More, Think Less (SMTL) is a framework for long-horizon agentic search that targets both efficiency and generalization.<n>We train an end-to-end agent using supervised fine-tuning and reinforcement learning, achieving strong and often state of the art performance across benchmarks.
  arXiv  Detail & Related papers  (2026-02-26T06:46:41Z)
• SE-Search: Self-Evolving Search Agent via Memory and Dense Reward [87.79131676521656]
  Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge.<n>Existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals.<n>We propose textbfSelf-textbfEvolving textbfSearch, a Self-Evolving Search agent that improves online search behavior through three components.
  arXiv  Detail & Related papers  (2026-02-06T09:14:07Z)
• Your Reward Function for RL is Your Best PRM for Search: Unifying RL and Search-Based TTS [62.22644307952087]
  We introduce AIRL-S, the first natural unification of RL-based and search-based TTS.<n>We leverage adversarial inverse reinforcement learning (AIRL) combined with group relative policy optimization (GRPO) to learn a dense, dynamic PRM directly from correct reasoning traces.<n>Our results show that our unified approach improves performance by 9 % on average over the base model, matching GPT-4o.
  arXiv  Detail & Related papers  (2025-08-19T23:41:15Z)
• R-Search: Empowering LLM Reasoning with Search via Multi-Reward Reinforcement Learning [0.8388591755871735]
  R-Search is a reinforcement learning framework for Reasoning-Search integration.<n>It guides large language models to autonomously execute multi-step reasoning with deep search interaction.<n>R-Search learns optimal reasoning search interaction trajectories via multi-reward signals.
  arXiv  Detail & Related papers  (2025-06-04T17:29:22Z)
• Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning [50.419872452397684]
  Search-R1 is an extension of reinforcement learning for reasoning frameworks.<n>It generates search queries during step-by-step reasoning with real-time retrieval.<n>It improves performance by 41% (Qwen2.5-7B) and 20% (Qwen2.5-3B) over various RAG baselines.
  arXiv  Detail & Related papers  (2025-03-12T16:26:39Z)
• Learning to Rank for Multiple Retrieval-Augmented Models through Iterative Utility Maximization [21.115495457454365]
  This paper investigates the design of a unified search engine to serve multiple retrieval-augmented generation (RAG) agents.<n>We introduce an iterative approach where the search engine generates retrieval results for the RAG agents and gathers feedback on the quality of the retrieved documents during an offline phase.<n>We adapt this to an online setting, allowing the search engine to refine its behavior based on real-time individual agents feedback.
  arXiv  Detail & Related papers  (2024-10-13T17:53:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.