Learning to Retrieve from Agent Trajectories

• URL: http://arxiv.org/abs/2604.04949v1
• Date: Mon, 30 Mar 2026 17:59:02 GMT
• Title: Learning to Retrieve from Agent Trajectories
• Abstract summary: We argue that retrieval models for agentic search should be trained directly from agent interaction data.<n>We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions.<n>Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
• Score: 72.8923565916533
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Information retrieval (IR) systems have traditionally been designed and trained for human users, with learning-to-rank methods relying heavily on large-scale human interaction logs such as clicks and dwell time. With the rapid emergence of large language model (LLM) powered search agents, however, retrieval is increasingly consumed by agents rather than human beings, and is embedded as a core component within multi-turn reasoning and action loops. In this setting, retrieval models trained under human-centric assumptions exhibit a fundamental mismatch with the way agents issue queries and consume results. In this work, we argue that retrieval models for agentic search should be trained directly from agent interaction data. We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions. Through a systematic analysis of search agent trajectories, we identify key behavioral signals that reveal document utility, including browsing actions, unbrowsed rejections, and post-browse reasoning traces. Guided by these insights, we propose LRAT, a simple yet effective framework that mines high-quality retrieval supervision from agent trajectories and incorporates relevance intensity through weighted optimization. Extensive experiments on both in-domain and out-of-domain deep research benchmarks demonstrate that retrievers trained with LRAT consistently improve evidence recall, end-to-end task success, and execution efficiency across diverse agent architectures and scales. Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.

Related papers

• ITER: Interaction-Aware Retrieval for Agentic Search [42.72882137602385]
  Iter is an agent interaction-aware dense retriever trained using agent trajectory learning signals.<n>It consistently outperforms the existing agent-trajectory-trained dense retriever, LRAT, achieving an average improvement of 7.5% on InfoSeek-Eval and 13.5% on BrowseComp-Plus.
  arXiv  Detail & Related papers  (2026-08-28T04:36:02Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search [5.481048072631037]
  We propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training of search agents.<n>We extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge.<n>Our approach achieves substantial performance gains and also exhibits strong cross-task and cross-algorithm generalization.
  arXiv  Detail & Related papers  (2026-04-09T11:44:44Z)
• SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents [35.45207852488779]
  We introduce a Tool-Memory based self-evolving agentic framework SEARL.<n>Unlike approaches that directly utilize interaction experiences, our method constructs a structured experience memory that integrates planning with execution.<n>We evaluate our framework on knowledge reasoning and mathematics tasks, demonstrating its effectiveness in achieving more practical and efficient learning.
  arXiv  Detail & Related papers  (2026-04-09T04:38:47Z)
• SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval [61.73547199231695]
  SubSearch is a specialized framework that shifts from outcome-only supervision to intermediate reward signals.<n> Experiments show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces.<n>SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering.
  arXiv  Detail & Related papers  (2026-04-08T13:09:47Z)
• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• Empowering RepoQA-Agent based on Reinforcement Learning Driven by Monte-carlo Tree Search [70.63903518295785]
  We introduce RepoSearch-R1, a novel agentic reinforcement learning framework driven by Monte-carlo Tree Search.<n>Based on RepoSearch-R1, we construct a RepoQA-Agent specifically designed for repository question-answering tasks.
  arXiv  Detail & Related papers  (2025-10-30T09:10:36Z)
• Beneficial Reasoning Behaviors in Agentic Search and Effective Post-training to Obtain Them [23.986035712600657]
  We propose a reasoning-driven pipeline to study effective reasoning behavior patterns in agentic search.<n>We identify four beneficial reasoning behaviors: Information Verification, Authority Evaluation, Adaptive Search, and Error Recovery.<n>We show that behavior priming yields over 35% gains in Llama3.2-3B and Qwen3-1.7B compared to directly training agentic search models with RL.
  arXiv  Detail & Related papers  (2025-10-08T00:20:35Z)
• ReSeek: A Self-Correcting Framework for Search Agents with Instructive Rewards [18.92867715736209]
  We propose ReSeek, a novel self-correcting framework for training search agents.<n>Our framework introduces a self-correction mechanism that empowers the agent to dynamically identify and recover from erroneous search paths.<n>To mitigate the risk of data contamination in existing datasets, we introduce FictionalHot.
  arXiv  Detail & Related papers  (2025-10-01T06:44:28Z)
• DynaSearcher: Dynamic Knowledge Graph Augmented Search Agent via Multi-Reward Reinforcement Learning [5.280613615397194]
  DynaSearcher is an innovative search agent enhanced by dynamic knowledge graphs and multi-reward reinforcement learning (RL)<n>We employ a multi-reward RL framework for fine-grained control over training objectives such as retrieval accuracy, efficiency, and response quality.<n> Experimental results demonstrate that our approach achieves state-of-the-art answer accuracy on six multi-hop question answering datasets.
  arXiv  Detail & Related papers  (2025-07-23T09:58:31Z)
• Deep Research Agents: A Systematic Examination And Roadmap [109.53237992384872]
  Deep Research (DR) agents are designed to tackle complex, multi-turn informational research tasks.<n>In this paper, we conduct a detailed analysis of the foundational technologies and architectural components that constitute DR agents.
  arXiv  Detail & Related papers  (2025-06-22T16:52:48Z)
• Adversarial Search and Tracking with Multiagent Reinforcement Learning in Sparsely Observable Environment [7.195547595036644]
  We study a search and tracking (S&T) problem where a team of dynamic search agents must collaborate to track an adversarial, evasive agent. This problem is challenging for both model-based searching and reinforcement learning (RL) methods since the adversary exhibits reactionary and deceptive evasive behaviors in a large space leading to sparse detections for the search agents. We propose a novel Multi-Agent RL (MARL) framework that leverages the estimated adversary location from our learnable filtering model.
  arXiv  Detail & Related papers  (2023-06-20T05:31:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.