Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought

• URL: http://arxiv.org/abs/2605.07123v1
• Date: Fri, 08 May 2026 01:56:38 GMT
• Title: Convergence and Emergence of In-Context Reinforcement Learning with Chain of Thought
• Abstract summary: In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context.<n>Recent empirical studies further demonstrate that Chain-of-Thought (CoT) generation can amplify this ICRL capability.
• Score: 21.241323360100548
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-context reinforcement learning (ICRL) refers to the ability of RL agents to adapt to new tasks at inference time without parameter updates by conditioning on additional context. Recent empirical studies further demonstrate that Chain-of-Thought (CoT) generation can amplify this ICRL capability. This paper is the first to provide a theoretical understanding on how CoT interacts with ICRL. We conduct our analysis in a policy evaluation setup with linear Transformer. We prove that with specific Transformer parameters, the CoT generation process is equivalent to repeatedly executing temporal difference learning updates. Additionally, we provide finite sample convergence analysis showing that the policy evaluation error decreases geometrically with CoT length and eventually saturates at a statistical floor determined by the context length. We also prove that the desired Transformer parameters are a global minimizer of the pretraining loss, providing a theoretical understanding on the empirical emergence of those parameters.

Related papers

• Adaptive Recurrent Message Passing for Test Time Computing on Graphs [80.56829418752513]
  We show that recurrent graph models can overcome the intrinsic mismatch between graph data and architectural designs.<n>We propose AdaR, an Adaptive Recurrent graph model, empowering flexible test-time computing on various downstream tasks without changing model parameters.
  arXiv  Detail & Related papers  (2026-06-21T12:14:11Z)
• Transformers Provably Implement In-Context Reinforcement Learning with Policy Improvement [23.24714915251971]
  We investigate the ability of transformers to perform in-context reinforcement learning.<n>We show that a linear self-attention transformer block can provably implement policy-improvement methods.<n>We establish the first convergence guarantee in the ICRL literature.
  arXiv  Detail & Related papers  (2026-05-07T06:50:49Z)
• Understanding In-Context Learning for Nonlinear Regression with Transformers: Attention as Featurizer [45.94152084965753]
  Pre-trained transformers are able to learn from examples provided as part of the prompt without any weight updates.<n>We study in-context learning (ICL) in the nonlinear regression setting.<n>We numerically validate the theory on synthetic regression tasks.
  arXiv  Detail & Related papers  (2026-05-06T17:42:07Z)
• When Does Context Help? Error Dynamics of Contextual Information in Large Language Models [64.88201012057822]
  We present a unified theoretical framework for analyzing the effect of arbitrary contextual information in large language models.<n>Our analysis characterizes contextual influence through output error dynamics.<n> Experiments across ICL, retrieval-augmented generation, and memory evolution validate our theory and motivate a principled context selection strategy.
  arXiv  Detail & Related papers  (2026-02-09T05:58:41Z)
• Structuring Value Representations via Geometric Coherence in Markov Decision Processes [9.312400001335659]
  We propose emphGCR-RL (Geometric Coherence Regularized Reinforcement Learning) that computes a sequence of super-poset refinements.<n>Two novel algorithms by Q-learning and by actor--critic are developed to efficiently realize these super-poset refinements.<n>We empirically evaluate GCR-RL in a range of tasks and demonstrate significant improvements in sample efficiency and stable performance over strong baselines.
  arXiv  Detail & Related papers  (2026-02-03T01:35:58Z)
• Towards Monotonic Improvement in In-Context Reinforcement Learning [18.67894044930047]
  In-Context Reinforcement Learning (ICRL) has emerged as a promising paradigm for developing agents that can rapidly adapt to new tasks.<n>Recent approaches train large sequence models on monotonic policy improvement data from online RL, aiming to a continue improved testing time performance.<n>We propose two methods for estimating Context Value at both training and testing time.
  arXiv  Detail & Related papers  (2025-09-27T09:42:19Z)
• Towards Provable Emergence of In-Context Reinforcement Learning [20.54963042400693]
  Some reinforcement learning (RL) agents can solve a wide range of new out-of-distribution tasks without parameter updates after pretraining on some task distribution.<n>When evaluated in a new task, instead of making parameter updates, the pretrained agent conditions its policy on additional input called the context.<n>This phenomenon is typically called in-context RL (ICRL)
  arXiv  Detail & Related papers  (2025-09-22T20:23:09Z)
• Provable In-Context Learning of Nonlinear Regression with Transformers [66.99048542127768]
  In-context learning (ICL) is the ability to perform unseen tasks using task specific prompts without updating parameters.<n>Recent research has actively explored the training dynamics behind ICL, with much of the focus on relatively simple tasks.<n>This paper investigates more complex nonlinear regression tasks, aiming to uncover how transformers acquire in-context learning capabilities.
  arXiv  Detail & Related papers  (2025-07-28T00:09:28Z)
• Interpreting Affine Recurrence Learning in GPT-style Transformers [54.01174470722201]
  In-context learning allows GPT-style transformers to generalize during inference without modifying their weights. This paper focuses specifically on their ability to learn and predict affine recurrences as an ICL task. We analyze the model's internal operations using both empirical and theoretical approaches.
  arXiv  Detail & Related papers  (2024-10-22T21:30:01Z)
• Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis [82.51626700527835]
  Chain-of-shift (CoT) is an efficient method that enables the reasoning ability of large language models by augmenting the query using examples with multiple intermediate steps.<n>We show that despite the theoretical success of CoT, it fails to provide an accurate generalization when CoT does.
  arXiv  Detail & Related papers  (2024-10-03T03:12:51Z)
• Parameter-Efficient Fine-Tuning for Continual Learning: A Neural Tangent Kernel Perspective [125.00228936051657]
  We introduce NTK-CL, a novel framework that eliminates task-specific parameter storage while adaptively generating task-relevant features.<n>By fine-tuning optimizable parameters with appropriate regularization, NTK-CL achieves state-of-the-art performance on established PEFT-CL benchmarks.
  arXiv  Detail & Related papers  (2024-07-24T09:30:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.