Retrieval-Augmented LLM Agents: Learning to Learn from Experience

• URL: http://arxiv.org/abs/2603.18272v1
• Date: Wed, 18 Mar 2026 20:45:04 GMT
• Title: Retrieval-Augmented LLM Agents: Learning to Learn from Experience
• Abstract summary: We study how to train retrieval-augmented LLM agents to leverage retrieved trajectories in-context.<n>We establish a robust supervised fine-tuning (SFT) recipe using LoRA that outperforms several state-of-the-art agent training pipelines.<n>Our results demonstrate that this combined approach significantly improves generalization to unseen tasks.
• Score: 16.248836438253814
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While large language models (LLMs) have advanced the development of general-purpose agents, achieving robust generalization to unseen tasks remains a significant challenge. Current approaches typically rely on either fine-tuning or training-free memory-augmented generation using retrieved experience; yet both have limitations: fine-tuning often fails to extrapolate to new tasks, while experience retrieval often underperforms compared to supervised baselines. In this work, we propose to combine these approaches and systematically study how to train retrieval-augmented LLM agents to effectively leverage retrieved trajectories in-context. First, we establish a robust supervised fine-tuning (SFT) recipe using LoRA that outperforms several state-of-the-art agent training pipelines. Second, we provide a detailed analysis of key design choices for experience retrieval, identifying optimal strategies for storage, querying, and trajectory selection. Finally, we propose a pipeline that integrates experience retrieval into the fine-tuning process. Our results demonstrate that this combined approach significantly improves generalization to unseen tasks, providing a scalable and effective framework for building agents that learn to learn from experience.

Related papers

• Transferable Expertise for Autonomous Agents via Real-World Case-Based Learning [6.277598389891074]
  We propose a case-based learning framework that converts experience from past tasks into reusable knowledge assets.<n>Unlike methods based mainly on pretrained knowledge or static prompts, our framework emphasizes extracting and reusing task-relevant knowledge.
  arXiv  Detail & Related papers  (2026-04-14T13:31:47Z)
• Beyond Stochastic Exploration: What Makes Training Data Valuable for Agentic Search [5.481048072631037]
  We propose a novel framework, Hierarchical Experience (HiExp), to enhance the performance and training of search agents.<n>We extract empirical knowledge through contrastive analysis and a multi-level clustering mechanism, transforming raw reasoning trajectories into hierarchical experience knowledge.<n>Our approach achieves substantial performance gains and also exhibits strong cross-task and cross-algorithm generalization.
  arXiv  Detail & Related papers  (2026-04-09T11:44:44Z)
• SEARL: Joint Optimization of Policy and Tool Graph Memory for Self-Evolving Agents [35.45207852488779]
  We introduce a Tool-Memory based self-evolving agentic framework SEARL.<n>Unlike approaches that directly utilize interaction experiences, our method constructs a structured experience memory that integrates planning with execution.<n>We evaluate our framework on knowledge reasoning and mathematics tasks, demonstrating its effectiveness in achieving more practical and efficient learning.
  arXiv  Detail & Related papers  (2026-04-09T04:38:47Z)
• Learning to Retrieve from Agent Trajectories [72.8923565916533]
  We argue that retrieval models for agentic search should be trained directly from agent interaction data.<n>We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions.<n>Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
  arXiv  Detail & Related papers  (2026-03-30T17:59:02Z)
• Experiential Reflective Learning for Self-Improving LLM Agents [1.1074589887824053]
  We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation.<n>ERL reflects on task trajectories and outcomes to generate actionable lessons that transfer across tasks.<n>ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability.
  arXiv  Detail & Related papers  (2026-03-25T11:43:22Z)
• APEX-Searcher: Augmenting LLMs' Search Capabilities through Agentic Planning and Execution [10.643859464982071]
  APEX-Searcher is a novel Agentic Planning and Execution framework to augment large language models (LLMs) search capabilities.<n>It first employs RL with decomposition-specific rewards to optimize strategic planning.<n>It then applies supervised fine-tuning on high-quality multi-hop trajectories to equip the model with robust iterative sub-task execution capabilities.
  arXiv  Detail & Related papers  (2026-03-14T09:17:12Z)
• LamRA: Large Multimodal Model as Your Advanced Retrieval Assistant [63.28378110792787]
  We introduce LamRA, a versatile framework designed to empower Large Multimodal Models with sophisticated retrieval and reranking capabilities.<n>For retrieval, we adopt a two-stage training strategy comprising language-only pre-training and multimodal instruction tuning.<n>For reranking, we employ joint training for both pointwise and listwise reranking, offering two distinct ways to further boost the retrieval performance.
  arXiv  Detail & Related papers  (2024-12-02T17:10:16Z)
• From Novice to Expert: LLM Agent Policy Optimization via Step-wise Reinforcement Learning [62.54484062185869]
  We introduce StepAgent, which utilizes step-wise reward to optimize the agent's reinforcement learning process.<n>We propose implicit-reward and inverse reinforcement learning techniques to facilitate agent reflection and policy adjustment.
  arXiv  Detail & Related papers  (2024-11-06T10:35:11Z)
• CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharing [70.25689961697523]
  We propose a generalizable algorithm that enhances sequential reasoning by cross-task experience sharing and selection. Our work bridges the gap between existing sequential reasoning paradigms and validates the effectiveness of leveraging cross-task experiences.
  arXiv  Detail & Related papers  (2024-10-22T03:59:53Z)
• Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinement [50.481380478458945]
  Iterative step-level Process Refinement (IPR) framework provides detailed step-by-step guidance to enhance agent training. Our experiments on three complex agent tasks demonstrate that our framework outperforms a variety of strong baselines.
  arXiv  Detail & Related papers  (2024-06-17T03:29:13Z)
• Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents [49.85633804913796]
  We present an exploration-based trajectory optimization approach, referred to as ETO. This learning method is designed to enhance the performance of open LLM agents. Our experiments on three complex tasks demonstrate that ETO consistently surpasses baseline performance by a large margin.
  arXiv  Detail & Related papers  (2024-03-04T21:50:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.