Training a Conditioned Video Game Agent on a VLM Annotated Dataset

• URL: http://arxiv.org/abs/2608.05954v1
• Date: Thu, 06 Aug 2026 12:25:54 GMT
• Title: Training a Conditioned Video Game Agent on a VLM Annotated Dataset
• Abstract summary: Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning.<n>We show that offline RL can then be used to train a conditioned agent that responds accordingly to the desired returns.
• Score: 3.102704962960118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning. In the specific case of video games, access to the game engine is required to get rewards for training (e.g. to collect rewards from the environment). Furthermore, the proper identification and weighting of the rewards generally requires a difficult trial-and-error approach. Lastly, rewards are often sparse and understanding how they eventually affect the learned policy is a non-trivial exercise. To ease these issues we propose annotating a video game dataset with Vision Language Models (VLMs) instructed to extract human defined rewards. We show that offline RL can then be used to train a conditioned agent that responds accordingly to the desired returns and we discuss the difficulties and limitations that emerged in our early experiments.

Related papers

• VLMs for Videogame Data Annotation [3.102704962960118]
  We show that Vision Language Models (VLMs) and Artificial Intelligence (AI) agents have revolutionized how engineers approach complex problems in real-world applications.<n>Their adoption in video games is on the other hand limited by the extreme variability of the synthetic scenarios and their poor compliance with realworld physics.
  arXiv  Detail & Related papers  (2026-08-06T12:20:53Z)
• Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning [89.93384726755106]
  Vision-language reinforcement learning (RL) has primarily focused on narrow domains.<n>We find video games inherently provide rich visual elements and mechanics that are easy to verify.<n>To fully use the multimodal and verifiable reward in video games, we propose Game-RL.
  arXiv  Detail & Related papers  (2025-05-20T03:47:44Z)
• ViVa: Video-Trained Value Functions for Guiding Online RL from Diverse Data [56.217490064597506]
  We propose and analyze a data-driven methodology that automatically guides RL by learning from widely available video data.<n>We use intent-conditioned value functions to learn from diverse videos and incorporate these goal-conditioned values into the reward.<n>Our experiments show that video-trained value functions work well with a variety of data sources, exhibit positive transfer from human video pre-training, can generalize to unseen goals, and scale with dataset size.
  arXiv  Detail & Related papers  (2025-03-23T21:24:33Z)
• SPRING: Studying the Paper and Reasoning to Play Games [102.5587155284795]
  We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM) In experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories.
  arXiv  Detail & Related papers  (2023-05-24T18:14:35Z)
• Video Prediction Models as Rewards for Reinforcement Learning [127.53893027811027]
  VIPER is an algorithm that leverages pretrained video prediction models as action-free reward signals for reinforcement learning. We see our work as starting point for scalable reward specification from unlabeled videos.
  arXiv  Detail & Related papers  (2023-05-23T17:59:33Z)
• Learning to Play Text-based Adventure Games with Maximum Entropy Reinforcement Learning [4.698846136465861]
  We adapt the soft-actor-critic (SAC) algorithm to the text-based environment. We show that the reward shaping technique helps the agent to learn the policy faster and achieve higher scores.
  arXiv  Detail & Related papers  (2023-02-21T15:16:12Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)
• Reinforcement Learning with Videos: Combining Offline Observations with Interaction [151.73346150068866]
  Reinforcement learning is a powerful framework for robots to acquire skills from experience. Videos of humans are a readily available source of broad and interesting experiences. We propose a framework for reinforcement learning with videos.
  arXiv  Detail & Related papers  (2020-11-12T17:15:48Z)
• Action Guidance: Getting the Best of Sparse Rewards and Shaped Rewards for Real-time Strategy Games [0.0]
  Training agents using Reinforcement Learning in games with sparse rewards is a challenging problem. We present a novel technique that successfully trains agents to eventually optimize the true objective in games with sparse rewards.
  arXiv  Detail & Related papers  (2020-10-05T03:43:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.