VLMs for Videogame Data Annotation

• URL: http://arxiv.org/abs/2608.05949v1
• Date: Thu, 06 Aug 2026 12:20:53 GMT
• Title: VLMs for Videogame Data Annotation
• Abstract summary: We show that Vision Language Models (VLMs) and Artificial Intelligence (AI) agents have revolutionized how engineers approach complex problems in real-world applications.<n>Their adoption in video games is on the other hand limited by the extreme variability of the synthetic scenarios and their poor compliance with realworld physics.
• Score: 3.102704962960118
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Language Models (VLMs) and Artificial Intelligence (AI) agents have revolutionized how engineers approach complex problems in real-world applications. Their adoption in video games is on the other hand limited by the extreme variability of the synthetic scenarios and their poor compliance with real-world physics. Here we investigate the use of VLMs for annotating video game frame sequences with reward signals, a task with several potential applications including, among others, conditioned training and offline reinforcement learning. We show that VLMs often struggle to answer basic questions on racing video games (although we observed a similar behavior on other game genres) and discuss countermeasures such as VLM output mixing and prompt optimization. We also show how input sequence length, resolution, and question batching affect the annotation quality and its token consumption.

Related papers

• Training a Conditioned Video Game Agent on a VLM Annotated Dataset [3.102704962960118]
  Reinforcement Learning (RL) is a powerful but far from easy-to-use technique for policy learning.<n>We show that offline RL can then be used to train a conditioned agent that responds accordingly to the desired returns.
  arXiv  Detail & Related papers  (2026-08-06T12:25:54Z)
• Self-Evolving Visual Questioner [53.949845616431624]
  Vision-language models (VLMs) are typically trained as passive answerers.<n>We show that a VLM can continuously improve itself as a visual questioner without any external supervision.<n>We propose a self-evolving framework that uses a VLM itself as both a proposer and a filter to produce harder, more informative, and visual-centric questions.
  arXiv  Detail & Related papers  (2026-06-11T21:45:46Z)
• Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs [53.851880753186585]
  We propose a unified incomplete video-language model to process the incomplete multi-modal inputs.<n>Our method can serve as a plug-and-play module for previous works to improve their performance in various multi-modal tasks.
  arXiv  Detail & Related papers  (2026-05-27T03:18:10Z)
• GameVerse: Can Vision-Language Models Learn from Video-based Reflection? [15.723982947450189]
  We present GameVerse, a comprehensive video game benchmark that enables a reflective visual interaction loop.<n>Moving beyond traditional fire-and-forget evaluations, it uses a novel reflect-and-retry paradigm to assess how Vision-Language Models internalize visual experience and improve policies.
  arXiv  Detail & Related papers  (2026-03-01T14:29:39Z)
• SIMPACT: Simulation-Enabled Action Planning using Vision-Language Models [60.80050275581661]
  Vision-Language Models (VLMs) exhibit remarkable common-sense and semantic reasoning capabilities.<n>They lack a grounded understanding of physical dynamics.<n>We present S, a test-time, SIMulation-enabled ACTion Planning framework.<n>Our method demonstrates state-of-the-art performance on five challenging, real-world rigid-body and deformable manipulation tasks.
  arXiv  Detail & Related papers  (2025-12-05T18:51:03Z)
• Thinking With Videos: Multimodal Tool-Augmented Reinforcement Learning for Long Video Reasoning [39.6349428129868]
  multimodal large language models (MLLMs) are crucial for downstream tasks like video question answering and temporal grounding.<n>We propose Video Intelligence via Tool-Augmented Learning (VITAL), a novel end-to-end agentic video reasoning framework.<n>With a visual toolbox, the model can densely sample new video frames on demand and generate multimodal CoT for precise long video reasoning.
  arXiv  Detail & Related papers  (2025-08-06T13:03:21Z)
• SIRI-Bench: Challenging VLMs' Spatial Intelligence through Complex Reasoning Tasks [51.774165536666864]
  We introduce SIRI-Bench, a benchmark designed to evaluate Vision-Language Models' structural spatial intelligence.<n>Bench comprises 9,000 video-question-answer triplets, where each problem is embedded in a realistic 3D scene.<n> Experimental results reveal that state-of-the-art VLMs struggle significantly on SIRI-Bench, underscoring the challenge of structural spatial reasoning.
  arXiv  Detail & Related papers  (2025-06-17T13:40:00Z)
• Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning [89.93384726755106]
  Vision-language reinforcement learning (RL) has primarily focused on narrow domains.<n>We find video games inherently provide rich visual elements and mechanics that are easy to verify.<n>To fully use the multimodal and verifiable reward in video games, we propose Game-RL.
  arXiv  Detail & Related papers  (2025-05-20T03:47:44Z)
• Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation [90.00687889213991]
  Solving complex long-horizon robotic manipulation problems requires sophisticated high-level planning capabilities.<n>Vision-language models (VLMs) pretrained on Internet data could in principle offer a framework for tackling such problems.<n>In this paper, we introduce a novel test-time framework that enhancesVLMs' physical reasoning capabilities for multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2025-02-23T20:42:15Z)
• Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case [20.14197375326218]
  This research aims to provide new insights and directions for applying multimodal agents in complex action game environments. We select an ARPG, Black Myth: Wukong'', as a research platform to explore the capability boundaries of existing vision language models. We will release a human operation dataset containing recorded gameplay videos and operation logs, including mouse and keyboard actions.
  arXiv  Detail & Related papers  (2024-09-19T16:30:25Z)
• ViCor: Bridging Visual Understanding and Commonsense Reasoning with Large Language Models [27.5219975853389]
  We find that pre-trained vision-and-language models (VLMs) and large language models (LLMs) are good at different kinds of visual commonsense reasoning problems. For problems where the goal is to infer conclusions beyond image content,VLMs face difficulties, while LLMs, given sufficient visual evidence, can use commonsense to infer the answer well.
  arXiv  Detail & Related papers  (2023-10-09T17:10:35Z)
• SPRING: Studying the Paper and Reasoning to Play Games [102.5587155284795]
  We propose a novel approach, SPRING, to read the game's original academic paper and use the knowledge learned to reason and play the game through a large language model (LLM) In experiments, we study the quality of in-context "reasoning" induced by different forms of prompts under the setting of the Crafter open-world environment. Our experiments suggest that LLMs, when prompted with consistent chain-of-thought, have great potential in completing sophisticated high-level trajectories.
  arXiv  Detail & Related papers  (2023-05-24T18:14:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.