Goal-Conditioned Agents that Learn Everything All at Once

• URL: http://arxiv.org/abs/2605.23551v1
• Date: Fri, 22 May 2026 12:17:09 GMT
• Title: Goal-Conditioned Agents that Learn Everything All at Once
• Abstract summary: All-goals learning, where each transition is used for learning off-policy with respect to every goal, allows agents to extract maximal information.<n>This can be overcome by jointly outputting values and actions for every goal at once, allowing for efficient, parallel all-goals updates.<n>We show that this approach significantly outperforms other methods on goal-conditioned Craftax.
• Score: 14.217160378270266
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A goal-conditioned reinforcement learning agent exploring an environment will see a wealth of information throughout a trajectory, most of which is discarded when only performing on-policy updates with respect to the commanded goal. All-goals learning, where each transition is used for learning off-policy with respect to every goal, allows agents to extract maximal information, however it is usually computationally infeasible when done via naive relabelling. This can be overcome by jointly outputting values and actions for every goal at once, allowing for efficient, parallel all-goals updates with a single pass through the network, in a process we call Learning Everything all at Once (LEO). We show that this approach significantly outperforms other methods on goal-conditioned Craftax and is competitive with existing baselines on continuous control environments, while achieving a >250x speed-up compared to all-goals relabelling. We then go on to show that this approach can be made even more powerful by using LEO as a teacher network, rather than a direct actor. We hope that, by unlocking all-goals learning at scale, LEO can serve as a useful tool for RL practitioners in complex environments. We open source our code.

Related papers

• ViVa: Video-Trained Value Functions for Guiding Online RL from Diverse Data [56.217490064597506]
  We propose and analyze a data-driven methodology that automatically guides RL by learning from widely available video data.<n>We use intent-conditioned value functions to learn from diverse videos and incorporate these goal-conditioned values into the reward.<n>Our experiments show that video-trained value functions work well with a variety of data sources, exhibit positive transfer from human video pre-training, can generalize to unseen goals, and scale with dataset size.
  arXiv  Detail & Related papers  (2025-03-23T21:24:33Z)
• An Invitation to Deep Reinforcement Learning [24.807012576054504]
  Training a deep neural network to maximize a target objective has become the standard recipe for successful machine learning.<n>Common objectives like intersection over union (IoU), bilingual evaluation understudy (BLEU) score or rewards cannot be optimized with supervised learning.<n>Reinforcement learning (RL) has emerged as a promising alternative for optimizing deep neural networks to maximize non-differentiable objectives.
  arXiv  Detail & Related papers  (2023-12-13T18:57:23Z)
• HIQL: Offline Goal-Conditioned RL with Latent States as Actions [81.67963770528753]
  We propose a hierarchical algorithm for goal-conditioned RL from offline data. We show how this hierarchical decomposition makes our method robust to noise in the estimated value function. Our method can solve long-horizon tasks that stymie prior methods, can scale to high-dimensional image observations, and can readily make use of action-free data.
  arXiv  Detail & Related papers  (2023-07-22T00:17:36Z)
• Robust Meta-Representation Learning via Global Label Inference and Classification [42.81340522184904]
  We introduce Meta Label Learning (MeLa), a novel meta-learning algorithm that learns task relations by inferring global labels across tasks. MeLa outperforms existing methods across a diverse range of benchmarks, in particular under a more challenging setting where the number of training tasks is limited and labels are task-specific.
  arXiv  Detail & Related papers  (2022-12-22T13:46:47Z)
• Goal-Conditioned Q-Learning as Knowledge Distillation [136.79415677706612]
  We explore a connection between off-policy reinforcement learning in goal-conditioned settings and knowledge distillation. We empirically show that this can improve the performance of goal-conditioned off-policy reinforcement learning when the space of goals is high-dimensional. We also show that this technique can be adapted to allow for efficient learning in the case of multiple simultaneous sparse goals.
  arXiv  Detail & Related papers  (2022-08-28T22:01:10Z)
• Learning user-defined sub-goals using memory editing in reinforcement learning [0.0]
  The aim of reinforcement learning (RL) is to allow the agent to achieve the final goal. I propose a methodology to achieve the user-defined sub-goals as well as the final goal using memory editing. I expect that this methodology can be used in the fields that need to control the agent in a variety of scenarios.
  arXiv  Detail & Related papers  (2022-05-01T05:19:51Z)
• Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learning [71.52722621691365]
  Building generalizable goal-conditioned agents from rich observations is a key to reinforcement learning (RL) solving real world problems. We propose a new form of state abstraction called goal-conditioned bisimulation. We learn this representation using a metric form of this abstraction, and show its ability to generalize to new goals in simulation manipulation tasks.
  arXiv  Detail & Related papers  (2022-04-27T17:00:11Z)
• PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learning [102.36450942613091]
  We propose an inverse reinforcement learning algorithm, called emphinverse temporal difference learning (ITD) We show how to seamlessly integrate ITD with learning from online environment interactions, arriving at a novel algorithm for reinforcement learning with demonstrations, called $Psi Phi$-learning.
  arXiv  Detail & Related papers  (2021-02-24T21:12:09Z)
• Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning [0.0]
  We show how a pre-trained task-independent language model can make a goal-conditional RL agent more sample efficient. We do this by facilitating transfer learning between different related tasks.
  arXiv  Detail & Related papers  (2020-07-10T06:42:00Z)
• Automatic Curriculum Learning through Value Disagreement [95.19299356298876]
  Continually solving new, unsolved tasks is the key to learning diverse behaviors. In the multi-task domain, where an agent needs to reach multiple goals, the choice of training goals can largely affect sample efficiency. We propose setting up an automatic curriculum for goals that the agent needs to solve. We evaluate our method across 13 multi-goal robotic tasks and 5 navigation tasks, and demonstrate performance gains over current state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-06-17T03:58:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.