Vintix: Action Model via In-Context Reinforcement Learning

• URL: http://arxiv.org/abs/2501.19400v1
• Date: Fri, 31 Jan 2025 18:57:08 GMT
• Title: Vintix: Action Model via In-Context Reinforcement Learning
• Authors: Andrey Polubarov, Nikita Lyubaykin, Alexander Derevyagin, Ilya Zisman, Denis Tarasov, Alexander Nikulin, Vladislav Kurenkov,
• Abstract summary: We present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning.<n>Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models.
• Score: 72.65703565352769
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In-Context Reinforcement Learning (ICRL) represents a promising paradigm for developing generalist agents that learn at inference time through trial-and-error interactions, analogous to how large language models adapt contextually, but with a focus on reward maximization. However, the scalability of ICRL beyond toy tasks and single-domain settings remains an open challenge. In this work, we present the first steps toward scaling ICRL by introducing a fixed, cross-domain model capable of learning behaviors through in-context reinforcement learning. Our results demonstrate that Algorithm Distillation, a framework designed to facilitate ICRL, offers a compelling and competitive alternative to expert distillation to construct versatile action models. These findings highlight the potential of ICRL as a scalable approach for generalist decision-making systems. Code to be released at https://github.com/dunnolab/vintix

Related papers

• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model [29.524164786422368]
  Recently, DeepSeek R1 has shown that reinforcement learning can substantially improve the reasoning capabilities of Large Language Models (LLMs) We investigate the extension of R1-style reinforcement learning to Vision-Language Models (VLMs) We develop VLM-R1, a dedicated framework designed to harness RL for improving VLMs' performance on general vision-language tasks.
  arXiv  Detail & Related papers  (2025-04-10T10:05:15Z)
• OpenVLThinker: An Early Exploration to Complex Vision-Language Reasoning via Iterative Self-Improvement [91.88062410741833]
  This study investigates whether similar reasoning capabilities can be successfully integrated into large vision-language models (LVLMs) We consider an approach that iteratively leverages supervised fine-tuning (SFT) on lightweight training data and Reinforcement Learning (RL) to further improve model generalization. OpenVLThinker, a LVLM exhibiting consistently improved reasoning performance on challenging benchmarks such as MathVista, MathVerse, and MathVision, demonstrates the potential of our strategy for robust vision-language reasoning.
  arXiv  Detail & Related papers  (2025-03-21T17:52:43Z)
• OmniRL: In-Context Reinforcement Learning by Large-Scale Meta-Training in Randomized Worlds [35.652208216209985]
  We introduce OmniRL, a highly generalizable in-context reinforcement learning model that is meta-trained on hundreds of thousands of diverse tasks. For the first time, we demonstrate that in-context learning (ICL) alone, without any gradient-based fine-tuning, can successfully tackle unseen Gymnasium tasks.
  arXiv  Detail & Related papers  (2025-02-05T03:59:13Z)
• On the Modeling Capabilities of Large Language Models for Sequential Decision Making [52.128546842746246]
  Large pretrained models are showing increasingly better performance in reasoning and planning tasks. We evaluate their ability to produce decision-making policies, either directly, by generating actions, or indirectly. In environments with unfamiliar dynamics, we explore how fine-tuning LLMs with synthetic data can significantly improve their reward modeling capabilities.
  arXiv  Detail & Related papers  (2024-10-08T03:12:57Z)
• LLMs Are In-Context Bandit Reinforcement Learners [30.192422586838997]
  Large Language Models (LLMs) excel at in-context learning (ICL), a supervised learning technique that relies on adding annotated examples to the model context.<n>We investigate a contextual bandit version of in-context reinforcement learning (ICRL), where models learn in-context, online, from external reward, instead of supervised data.
  arXiv  Detail & Related papers  (2024-10-07T17:45:00Z)
• Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement [67.1393112206885]
  Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. We introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks.
  arXiv  Detail & Related papers  (2024-02-09T07:45:26Z)
• Learning by Doing: An Online Causal Reinforcement Learning Framework with Causal-Aware Policy [38.86867078596718]
  We consider explicitly modeling the generation process of states with the graphical causal model. We formulate the causal structure updating into the RL interaction process with active intervention learning of the environment.
  arXiv  Detail & Related papers  (2024-02-07T14:09:34Z)
• A Neuromorphic Architecture for Reinforcement Learning from Real-Valued Observations [0.34410212782758043]
  Reinforcement Learning (RL) provides a powerful framework for decision-making in complex environments. This paper presents a novel Spiking Neural Network (SNN) architecture for solving RL problems with real-valued observations.
  arXiv  Detail & Related papers  (2023-07-06T12:33:34Z)
• Large Language Models can Implement Policy Iteration [18.424558160071808]
  In-Context Policy Iteration is an algorithm for performing Reinforcement Learning (RL), in-context, using foundation models. ICPI learns to perform RL tasks without expert demonstrations or gradients. ICPI iteratively updates the contents of the prompt from which it derives its policy through trial-and-error interaction with an RL environment.
  arXiv  Detail & Related papers  (2022-10-07T21:18:22Z)
• INFOrmation Prioritization through EmPOWERment in Visual Model-Based RL [90.06845886194235]
  We propose a modified objective for model-based reinforcement learning (RL) We integrate a term inspired by variational empowerment into a state-space model based on mutual information. We evaluate the approach on a suite of vision-based robot control tasks with natural video backgrounds.
  arXiv  Detail & Related papers  (2022-04-18T23:09:23Z)
• Jump-Start Reinforcement Learning [68.82380421479675]
  We present a meta algorithm that can use offline data, demonstrations, or a pre-existing policy to initialize an RL policy. In particular, we propose Jump-Start Reinforcement Learning (JSRL), an algorithm that employs two policies to solve tasks. We show via experiments that JSRL is able to significantly outperform existing imitation and reinforcement learning algorithms.
  arXiv  Detail & Related papers  (2022-04-05T17:25:22Z)
• Contextualize Me -- The Case for Context in Reinforcement Learning [49.794253971446416]
  Contextual Reinforcement Learning (cRL) provides a framework to model such changes in a principled manner. We show how cRL contributes to improving zero-shot generalization in RL through meaningful benchmarks and structured reasoning about generalization tasks.
  arXiv  Detail & Related papers  (2022-02-09T15:01:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.