Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning

• URL: http://arxiv.org/abs/2605.00347v1
• Date: Fri, 01 May 2026 02:05:56 GMT
• Title: Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning
• Abstract summary: In this work, we study RL-based training of vision-language models (VLMs) for long-horizon decision-making in Super Mario Land.<n>We propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency.<n>We introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game.
• Score: 50.464623632604976
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given the rapidly growing capabilities of vision-language models (VLMs), extending them to interactive decision-making tasks such as video games has emerged as a promising frontier. However, existing approaches either rely on large-scale supervised fine-tuning (SFT) on human trajectories or apply reinforcement learning (RL) only in relatively short-horizon settings (typically around 20--30 turns). In this work, we study RL-based training of VLMs for long-horizon decision-making in Super Mario Land, a visually grounded environment requiring 100+ turns of interaction with coordinated perception, reasoning, and action. We begin with a systematic investigation of key algorithmic components and propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency over critic-free methods such as GRPO and Reinforce++. We further show that pretrained VLMs provide strong action priors, significantly improving sample efficiency during RL training and reducing the need for manual design choices such as action engineering, compared to classical deep RL trained from scratch. Building on these insights, we introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game and at least 3 times average game progresses than frontier models. Moreover, the trained models exhibit consistent improvements under both in-game and cross-game generalization settings, while maintaining general-domain capabilities. Overall, our results identify key ingredients for making RL stable and effective in long-horizon, multi-modal settings, and provide practical guidance for developing VLMs as embodied agents.

Related papers

• VLM-Guided Experience Replay [41.08659748023147]
  We propose to use Vision-Language Models (VLMs) to guide the prioritization of experiences in the replay buffer.<n>Our key idea is to use a frozen, pre-trained VLM as an automated evaluator to identify and prioritize promising sub-trajectories from the agent's experiences.<n>Across scenarios, including game-playing and robotics, agents trained with our proposed prioritization method achieve 11-52% higher average success rates and improve sample efficiency by 19-45% compared to previous approaches.
  arXiv  Detail & Related papers  (2026-02-02T10:19:59Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learning [129.44038804430542]
  We introduce AgentGym-RL, a new framework to train LLM agents for multi-turn interactive decision-making through RL.<n>We propose ScalingInter-RL, a training approach designed for exploration-exploitation balance and stable RL optimization.<n>Our agents match or surpass commercial models on 27 tasks across diverse environments.
  arXiv  Detail & Related papers  (2025-09-10T16:46:11Z)
• ViaRL: Adaptive Temporal Grounding via Visual Iterated Amplification Reinforcement Learning [68.76048244253582]
  We introduce ViaRL, the first framework to leverage rule-based reinforcement learning (RL) for optimizing frame selection in video understanding.<n>ViaRL utilizes the answer accuracy of a downstream model as a reward signal to train a frame selector through trial-and-error.<n>ViaRL consistently delivers superior temporal grounding performance and robust generalization across diverse video understanding tasks.
  arXiv  Detail & Related papers  (2025-05-21T12:29:40Z)
• G1: Bootstrapping Perception and Reasoning Abilities of Vision-Language Model via Reinforcement Learning [37.18982308118744]
  Vision-Language Models (VLMs) excel in many direct multimodal tasks but struggle to translate this prowess into effective decision-making within interactive, visually rich environments like games.<n>We introduce VLM-Gym, a curated reinforcement learning environment featuring diverse visual games with unified interfaces and adjustable, compositional difficulty.<n>We train G0 models using pure RL-driven self-evolution, which demonstrate emergent perception and reasoning patterns.<n>Our resulting G1 models consistently surpass their teacher across all games and outperform leading proprietary models like Claude-3.7-Sonnet-Thinking.
  arXiv  Detail & Related papers  (2025-05-19T17:54:39Z)
• RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning [125.96848846966087]
  Training large language models (LLMs) as interactive agents presents unique challenges.<n>While reinforcement learning has enabled progress in static tasks, multi-turn agent RL training remains underexplored.<n>We propose StarPO, a general framework for trajectory-level agent RL, and introduce RAGEN, a modular system for training and evaluating LLM agents.
  arXiv  Detail & Related papers  (2025-04-24T17:57:08Z)
• Mastering the Unsupervised Reinforcement Learning Benchmark from Pixels [112.63440666617494]
  Reinforcement learning algorithms can succeed but require large amounts of interactions between the agent and the environment. We propose a new method to solve it, using unsupervised model-based RL, for pre-training the agent. We show robust performance on the Real-Word RL benchmark, hinting at resiliency to environment perturbations during adaptation.
  arXiv  Detail & Related papers  (2022-09-24T14:22:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.