LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure

• URL: http://arxiv.org/abs/2606.15306v1
• Date: Sat, 13 Jun 2026 13:51:22 GMT
• Title: LatentGym: A Testbed For Cross-Task Experiential Learning With Controllable Latent Structure
• Abstract summary: Cross-task experiential learning is pivotal in domains such as personalization and interactive assistance.<n>Existing training/evaluation frameworks do not provide shared, controllable latent structures and cannot measure whether or why agents improve.<n>We introduce LatentGym: a controllable suite in which each environment is organized around a ground-truth latent variable governing the structure across tasks.
• Score: 9.42262449979372
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We envision continually learning agentic systems that become more useful over time: as they encounter sequences of related tasks, they should infer the hidden structure shared across those tasks and use it to improve future decisions. This cross-task experiential learning capability is pivotal in domains such as personalization and interactive assistance, but existing training/evaluation frameworks do not provide shared, controllable latent structures and cannot measure whether or why agents improve. We introduce LatentGym: a controllable suite in which each environment is organized around a ground-truth latent variable governing the structure across tasks. Our construction yields metrics that separate exploration (whether the agent's actions gather information about the latent) from exploitation (whether the agent uses what it has gathered). We demonstrate our suite on empirical studies addressing three questions: how and why frontier models fail to adapt across related tasks; whether post-training on related task sequences improves general cross-task adaptation, and where those gains come from; and how design choices such as inter-task feedback shape training dynamics and generalization. Together, these results establish a controlled foundation for studying how LLM agents learn from experience across tasks, and for designing agents that adapt more reliably in sequential, personalized, and interactive settings.

Related papers

• From Question Answering to Task Completion: A Survey on Agent System and Harness Design [67.8241530947325]
  This survey examines LLM-based agents through a model-harness lens.<n>We first clarify the functional definition of agents and the implementation view of an LLM-based agent as a foundation model and an execution harness.<n>We then analyze the limits of model-centric scaling, trace four paradigms of agent engineering, and decompose the execution harness into six coupled runtime responsibilities.
  arXiv  Detail & Related papers  (2026-06-14T05:40:16Z)
• Look Before You Leap: Autonomous Exploration for LLM Agents [46.001025916022066]
  Large language model based agents often fail in unfamiliar environments due to premature exploitation.<n>We identify autonomous exploration as a critical yet underexplored capability for building adaptive agents.
  arXiv  Detail & Related papers  (2026-05-15T16:24:16Z)
• Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systems [65.90791804095561]
  Multi-agent systems address this through structured collaboration among specialized agents, but tighter coordination amplifies a less explored risk.<n>Existing surveys cover individual agent capabilities, multi-agent collaboration, or agent self-evolution separately, leaving the causal dependencies among them unexamined.<n>This survey provides a unified review organized around four causally linked stages, which we term the LIFE progression: Lay the capability foundation, Integrate agents through collaboration, Find faults through attribution, and Evolve through autonomous self-improvement.
  arXiv  Detail & Related papers  (2026-05-14T14:36:13Z)
• Scaling Environments for LLM Agents in the Era of Learning from Interaction: A Survey [30.673419015614233]
  A growing consensus is that agents should interact directly with environments and learn from experience through reinforcement learning.<n>We formalize this iterative process as the Generation-Execution-Feedback (GEF) loop, where environments generate tasks to challenge agents, return observations in response to agents' actions during task execution, and provide evaluative feedback on rollouts for subsequent learning.<n>Under this paradigm, environments function as indispensable producers of experiential data, highlighting the need to scale them toward greater complexity, realism, and interactivity.
  arXiv  Detail & Related papers  (2025-11-12T12:56:25Z)
• Cross-Task Experiential Learning on LLM-based Multi-Agent Collaboration [63.90193684394165]
  We introduce multi-agent cross-task experiential learning (MAEL), a novel framework that endows LLM-driven agents with explicit cross-task learning and experience accumulation.<n>During the experiential learning phase, we quantify the quality for each step in the task-solving workflow and store the resulting rewards.<n>During inference, agents retrieve high-reward, task-relevant experiences as few-shot examples to enhance the effectiveness of each reasoning step.
  arXiv  Detail & Related papers  (2025-05-29T07:24:37Z)
• A Survey on Complex Tasks for Goal-Directed Interactive Agents [60.53915548970061]
  This survey compiles relevant tasks and environments for evaluating goal-directed interactive agents. An up-to-date compilation of relevant resources can be found on our project website.
  arXiv  Detail & Related papers  (2024-09-27T08:17:53Z)
• Build generally reusable agent-environment interaction models [28.577502598559988]
  This paper tackles the problem of how to pre-train a model and make it generally reusable backbones for downstream task learning. We propose a method that builds an agent-environment interaction model by learning domain invariant successor features from the agent's vast experiences covering various tasks, then discretize them into behavior prototypes. We provide preliminary results that show downstream task learning based on a pre-trained embodied set structure can handle unseen changes in task objectives, environmental dynamics and sensor modalities.
  arXiv  Detail & Related papers  (2022-11-13T07:33:14Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Learning to Relate Depth and Semantics for Unsupervised Domain Adaptation [87.1188556802942]
  We present an approach for encoding visual task relationships to improve model performance in an Unsupervised Domain Adaptation (UDA) setting. We propose a novel Cross-Task Relation Layer (CTRL), which encodes task dependencies between the semantic and depth predictions. Furthermore, we propose an Iterative Self-Learning (ISL) training scheme, which exploits semantic pseudo-labels to provide extra supervision on the target domain.
  arXiv  Detail & Related papers  (2021-05-17T13:42:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.