Towards Provable Emergence of In-Context Reinforcement Learning

• URL: http://arxiv.org/abs/2509.18389v2
• Date: Fri, 03 Oct 2025 13:29:10 GMT
• Title: Towards Provable Emergence of In-Context Reinforcement Learning
• Authors: Jiuqi Wang, Rohan Chandra, Shangtong Zhang,
• Abstract summary: Some reinforcement learning (RL) agents can solve a wide range of new out-of-distribution tasks without parameter updates after pretraining on some task distribution.<n>When evaluated in a new task, instead of making parameter updates, the pretrained agent conditions its policy on additional input called the context.<n>This phenomenon is typically called in-context RL (ICRL)
• Score: 20.54963042400693
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Typically, a modern reinforcement learning (RL) agent solves a task by updating its neural network parameters to adapt its policy to the task. Recently, it has been observed that some RL agents can solve a wide range of new out-of-distribution tasks without parameter updates after pretraining on some task distribution. When evaluated in a new task, instead of making parameter updates, the pretrained agent conditions its policy on additional input called the context, e.g., the agent's interaction history in the new task. The agent's performance increases as the information in the context increases, with the agent's parameters fixed. This phenomenon is typically called in-context RL (ICRL). The pretrained parameters of the agent network enable the remarkable ICRL phenomenon. However, many ICRL works perform the pretraining with standard RL algorithms. This raises the central question this paper aims to address: Why can the RL pretraining algorithm generate network parameters that enable ICRL? We hypothesize that the parameters capable of ICRL are minimizers of the pretraining loss. This work provides initial support for this hypothesis through a case study. In particular, we prove that when a Transformer is pretrained for policy evaluation, one of the global minimizers of the pretraining loss can enable in-context temporal difference learning.

Related papers

• Just-In-Time Reinforcement Learning: Continual Learning in LLM Agents Without Gradient Updates [53.3717573880076]
  We introduce Just-In-Time Reinforcement Learning (JitRL), a training-free framework that enables test-time policy optimization without any gradient updates.<n>JitRL maintains a dynamic, non-parametric memory of experiences and retrieves relevant trajectories to estimate action advantages on-the-fly.<n>Experiments on WebArena and Jericho demonstrate that JitRL establishes a new state-of-the-art among training-free methods.
  arXiv  Detail & Related papers  (2026-01-26T14:16:51Z)
• Behavior Knowledge Merge in Reinforced Agentic Models [48.89546963456286]
  Reinforcement learning is central to post-training, particularly for agentic models that require specialized reasoning behaviors.<n>Existing merging methods are designed for supervised fine-tuning (SFT), and they are suboptimal to preserve task-specific capabilities on RL-trained agentic models.<n>We propose Reinforced Agent Merging (RAM), a distribution-aware merging framework explicitly designed for RL-trained agentic models.
  arXiv  Detail & Related papers  (2026-01-20T03:56:53Z)
• Safe In-Context Reinforcement Learning [19.11554873825587]
  In-context reinforcement learning (ICRL) is an emerging RL paradigm where the agent, after some pretraining procedure, is able to adapt to out-of-distribution test tasks without any parameter updates.<n>We propose the first method that promotes the safety of ICRL's adaptation process in the framework of constrained Markov Decision Processes.
  arXiv  Detail & Related papers  (2025-09-29T23:07:32Z)
• A Survey of In-Context Reinforcement Learning [41.74105124619678]
  Some agents can solve new tasks without updating any parameters by simply conditioning on additional context.<n>This paper surveys work on such behavior, known as in-context reinforcement learning.
  arXiv  Detail & Related papers  (2025-02-11T21:52:19Z)
• Transformers Can Learn Temporal Difference Methods for In-Context Reinforcement Learning [17.714908233024847]
  reinforcement learning (RL) agents learn to solve new tasks by updating their neural network parameters through interactions with the task environment.<n>Recent works demonstrate that some RL agents, after certain pretraining procedures, can learn to solve unseen new tasks without parameter updates.
  arXiv  Detail & Related papers  (2024-05-22T17:38:16Z)
• Hypernetworks for Zero-shot Transfer in Reinforcement Learning [21.994654567458017]
  Hypernetworks are trained to generate behaviors across a range of unseen task conditions. This work relates to meta RL, contextual RL, and transfer learning. Our method demonstrates significant improvements over baselines from multitask and meta RL approaches.
  arXiv  Detail & Related papers  (2022-11-28T15:48:35Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)
• Lean Evolutionary Reinforcement Learning by Multitasking with Importance Sampling [20.9680985132322]
  We introduce a novel neuroevolutionary multitasking (NuEMT) algorithm to transfer information from a set of auxiliary tasks to the target (full length) RL task. We demonstrate that the NuEMT algorithm data-lean evolutionary RL, reducing expensive agent-environment interaction data requirements.
  arXiv  Detail & Related papers  (2022-03-21T10:06:16Z)
• Retrieval-Augmented Reinforcement Learning [63.32076191982944]
  We train a network to map a dataset of past experiences to optimal behavior. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context. We show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores.
  arXiv  Detail & Related papers  (2022-02-17T02:44:05Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)
• What is Going on Inside Recurrent Meta Reinforcement Learning Agents? [63.58053355357644]
  Recurrent meta reinforcement learning (meta-RL) agents are agents that employ a recurrent neural network (RNN) for the purpose of "learning a learning algorithm" We shed light on the internal working mechanisms of these agents by reformulating the meta-RL problem using the Partially Observable Markov Decision Process (POMDP) framework.
  arXiv  Detail & Related papers  (2021-04-29T20:34:39Z)
• Off-Policy Meta-Reinforcement Learning Based on Feature Embedding Spaces [14.029933823101084]
  We propose a novel off-policy meta-RL method, embedding learning and evaluation of uncertainty (ELUE) ELUE learns a belief model over the embedding space and a belief-conditional policy and Q-function. We demonstrate that ELUE outperforms state-of-the-art meta RL methods through experiments on meta-RL benchmarks.
  arXiv  Detail & Related papers  (2021-01-06T05:51:38Z)
• Robust Deep Reinforcement Learning through Adversarial Loss [74.20501663956604]
  Recent studies have shown that deep reinforcement learning agents are vulnerable to small adversarial perturbations on the agent's inputs. We propose RADIAL-RL, a principled framework to train reinforcement learning agents with improved robustness against adversarial attacks.
  arXiv  Detail & Related papers  (2020-08-05T07:49:42Z)
• Dynamics Generalization via Information Bottleneck in Deep Reinforcement Learning [90.93035276307239]
  We propose an information theoretic regularization objective and an annealing-based optimization method to achieve better generalization ability in RL agents. We demonstrate the extreme generalization benefits of our approach in different domains ranging from maze navigation to robotic tasks. This work provides a principled way to improve generalization in RL by gradually removing information that is redundant for task-solving.
  arXiv  Detail & Related papers  (2020-08-03T02:24:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.