Training Language Agents to Learn from Experience

• URL: http://arxiv.org/abs/2605.20477v1
• Date: Tue, 19 May 2026 20:41:12 GMT
• Title: Training Language Agents to Learn from Experience
• Abstract summary: We introduce the In-context Training task, a framework for evaluating cross-task self-improvement in language agents.<n>We then propose an RL-based training pipeline for learning such reflections directly from experience.<n>Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families.
• Score: 25.31745060337765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language agents can adapt from experience in interactive environments, but current reflection-based methods can only self-correct within a single task instance. Whether such experience can be distilled into reusable lessons that improve performance on future unseen tasks remains unclear. We address this problem by introducing the In-context Training (ICT) task, a framework for evaluating cross-task self-improvement in language agents. In ICT, a reflector model observes trajectories collected by an actor model and generates system prompts intended to improve the actor's performance on future unseen tasks. We then propose an RL-based training pipeline for learning such reflections directly from experience, without human-provided examples. Across ALFWorld and MiniHack, our trained reflectors outperform an untrained baseline on most held-out task families, showing that the ability to learn from experience can itself be learned. In some cases, we observe generalisation beyond the benchmark on which the reflector was trained, to substantially different environments. Finally, we introduce MetaGym, a generic Python library for constructing meta-environments, enabling future research on self-improving language agents.

Related papers

• Evo-Harness: Context-to-Harness Skill Compilation for Self-Evolving Agents [67.97365535829098]
  Learning from experience is critical for developing capable, self-improving large language model (LLM) agents.<n>We propose Evo-Harness, which distills noisy, single-shot executions into reusable skill harnesses for cross-domain and topic-level adaptation.<n>Our analysis demonstrates the effectiveness of Evo-Harness and provides a principled understanding of how LLM agents can effectively learn on the fly.
  arXiv  Detail & Related papers  (2026-08-15T06:43:56Z)
• Experiential Reflective Learning for Self-Improving LLM Agents [1.1074589887824053]
  We introduce Experiential Reflective Learning (ERL), a simple self-improvement framework that enables rapid environment adaptation.<n>ERL reflects on task trajectories and outcomes to generate actionable lessons that transfer across tasks.<n>ERL improves success rate by 7.8% over a ReAct baseline, with large gains in task completion reliability.
  arXiv  Detail & Related papers  (2026-03-25T11:43:22Z)
• LANPO: Bootstrapping Language and Numerical Feedback for Reinforcement Learning in LLMs [73.27182315028021]
  LANPO is a framework that cleanly separates the roles of feedback: language guides exploration, while numerical rewards drive optimization.<n>Our work provides a robust method for integrating historical experiences into the LLM RL loop, creating more effective and data-efficient learning agents.
  arXiv  Detail & Related papers  (2025-10-18T15:51:19Z)
• Agent Learning via Early Experience [93.83579011718858]
  A long-term goal of language agents is to learn and improve through their own experience, ultimately outperforming humans in complex, real-world tasks.<n>Most current agents rely on supervised fine-tuning on expert data, which is challenging to scale and generalizes poorly.<n>We study two strategies of using such data: (1) Implicit world modeling, which uses collected states to ground the policy in environment dynamics; and (2) Self-reflection, where the agent learns from its suboptimal actions to improve reasoning and decision-making.
  arXiv  Detail & Related papers  (2025-10-09T17:59:17Z)
• Retrospex: Language Agent Meets Offline Reinforcement Learning Critic [4.776906435812746]
  Retrospex is an agent framework that analyzes past experiences in depth.<n>It combines the LLM's action likelihood with action values estimated by a Reinforcement Learning Critic.<n>We evaluate Retrospex in ScienceWorld, ALFWorld and Webshop environments.
  arXiv  Detail & Related papers  (2025-05-17T03:28:24Z)
• Playpen: An Environment for Exploring Learning Through Conversational Interaction [84.0413820245725]
  We investigate whether Dialogue Games can also serve as a source of feedback signals for learning.<n>We introduce Playpen, an environment for off- and online learning through Dialogue Game self-play.<n>We find that imitation learning through SFT improves performance on unseen instances, but negatively impacts other skills.
  arXiv  Detail & Related papers  (2025-04-11T14:49:33Z)
• Re-ReST: Reflection-Reinforced Self-Training for Language Agents [101.22559705696885]
  Self-training in language agents can generate supervision from the agent itself.<n>We present Reflection-Reinforced Self-Training (Re-ReST), which uses a textitreflector to refine low-quality generated samples.
  arXiv  Detail & Related papers  (2024-06-03T16:21:38Z)
• Vision-Language Models Provide Promptable Representations for Reinforcement Learning [67.40524195671479]
  We propose a novel approach that uses the vast amounts of general and indexable world knowledge encoded in vision-language models (VLMs) pre-trained on Internet-scale data for embodied reinforcement learning (RL) We show that our approach can use chain-of-thought prompting to produce representations of common-sense semantic reasoning, improving policy performance in novel scenes by 1.5 times.
  arXiv  Detail & Related papers  (2024-02-05T00:48:56Z)
• Reflexion: Language Agents with Verbal Reinforcement Learning [44.85337947858337]
  Reflexion is a novel framework to reinforce language agents not by updating weights, but through linguistic feedback. It is flexible enough to incorporate various types (scalar values or free-form language) and sources (external or internally simulated) of feedback signals. For example, Reflexion achieves a 91% pass@1 accuracy on the HumanEval coding benchmark, surpassing the previous state-of-the-art GPT-4 that achieves 80%.
  arXiv  Detail & Related papers  (2023-03-20T18:08:50Z)
• Curriculum Learning for Vision-and-Language Navigation [16.695511663714214]
  Vision-and-Language Navigation (VLN) is a task where an agent navigates in an embodied indoor environment under human instructions. Previous works ignore the distribution of sample difficulty and we argue that this potentially degrade their agent performance. We propose a novel curriculum-based training paradigm for VLN tasks that can balance human prior knowledge and agent learning progress.
  arXiv  Detail & Related papers  (2021-11-14T03:02:07Z)
• Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning [0.0]
  We show how a pre-trained task-independent language model can make a goal-conditional RL agent more sample efficient. We do this by facilitating transfer learning between different related tasks.
  arXiv  Detail & Related papers  (2020-07-10T06:42:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.