Do Vision Language Models Understand Human Engagement in Games?

• URL: http://arxiv.org/abs/2603.18480v1
• Date: Thu, 19 Mar 2026 04:32:12 GMT
• Title: Do Vision Language Models Understand Human Engagement in Games?
• Abstract summary: Inferring human engagement from gameplay video is important for game design and player-experience research.<n>We evaluate three vision-language models (VLMs) under six prompting strategies across nine first-person shooter games.<n>Results show that zero-shot VLM predictions are generally weak and often fail to outperform simple per-game majority-class baselines.
• Score: 5.398980203118197
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Inferring human engagement from gameplay video is important for game design and player-experience research, yet it remains unclear whether vision--language models (VLMs) can infer such latent psychological states from visual cues alone. Using the GameVibe Few-Shot dataset across nine first-person shooter games, we evaluate three VLMs under six prompting strategies, including zero-shot prediction, theory-guided prompts grounded in Flow, GameFlow, Self-Determination Theory, and MDA, and retrieval-augmented prompting. We consider both pointwise engagement prediction and pairwise prediction of engagement change between consecutive windows. Results show that zero-shot VLM predictions are generally weak and often fail to outperform simple per-game majority-class baselines. Memory- or retrieval-augmented prompting improves pointwise prediction in some settings, whereas pairwise prediction remains consistently difficult across strategies. Theory-guided prompting alone does not reliably help and can instead reinforce surface-level shortcuts. These findings suggest a perception--understanding gap in current VLMs: although they can recognize visible gameplay cues, they still struggle to robustly infer human engagement across games.

Related papers

• MIRL: Mutual Information-Guided Reinforcement Learning for Vision-Language Models [46.54440573184562]
  Reinforcement Learning with Verifiable Rewards (RLVR) offers a promising solution by optimizing policies using answer correctness signals.<n>We introduce MIRL, a decoupled framework that addresses both limitations by leveraging mutual information (MI) between generated descriptions and visual inputs as a cheap pre-screening signal.<n>Experiments on six vision-language reasoning benchmarks demonstrate that MIRL achieves 70.22% average accuracy.
  arXiv  Detail & Related papers  (2026-05-02T16:21:31Z)
• Beyond Survival: Evaluating LLMs in Social Deduction Games with Human-Aligned Strategies [54.08697738311866]
  Social deduction games like Werewolf combine language, reasoning, and strategy.<n>We curate a high-quality, human-verified multimodal Werewolf dataset containing over 100 hours of video, 32.4M utterance tokens, and 15 rule variants.<n>We propose a novel strategy-alignment evaluation that leverages the winning faction's strategies as ground truth in two stages.
  arXiv  Detail & Related papers  (2025-10-13T13:33:30Z)
• Self-Rewarding Vision-Language Model via Reasoning Decomposition [49.784411666601905]
  Vision-Language Models (VLMs) often suffer from visual hallucinations, saying things that are not actually in the image, and language shortcuts.<n>We introduce Vision-SR1, a self-rewarding method that improves visual reasoning without relying on external visual supervisions.<n>Our experiments demonstrate that Vision-SR1 improves visual reasoning, mitigates visual hallucinations, and reduces reliance on language shortcuts.
  arXiv  Detail & Related papers  (2025-08-27T08:01:03Z)
• VideoGameBench: Can Vision-Language Models complete popular video games? [8.5302862604852]
  Video games are crafted to be intuitive for humans to learn and master by leveraging innate inductive biases.<n>We introduce VideoGameBench, a benchmark consisting of 10 popular video games from the 1990s that VLMs directly interact with in real-time.<n>We show that frontier vision-language models struggle to progress beyond the beginning of each game.
  arXiv  Detail & Related papers  (2025-05-23T17:43:27Z)
• lmgame-Bench: How Good are LLMs at Playing Games? [60.01834131847881]
  We study the major challenges in using popular video games to evaluate modern large language model (LLM) agents.<n>We introduce lmgame-Bench to turn games into reliable evaluations.
  arXiv  Detail & Related papers  (2025-05-21T06:02:55Z)
• Game-RL: Synthesizing Multimodal Verifiable Game Data to Boost VLMs' General Reasoning [89.93384726755106]
  Vision-language reinforcement learning (RL) has primarily focused on narrow domains.<n>We find video games inherently provide rich visual elements and mechanics that are easy to verify.<n>To fully use the multimodal and verifiable reward in video games, we propose Game-RL.
  arXiv  Detail & Related papers  (2025-05-20T03:47:44Z)
• CognitionNet: A Collaborative Neural Network for Play Style Discovery in Online Skill Gaming Platform [6.665636945186558]
  We propose a two stage deep neural network, CognitionNet.<n>The first stage focuses on mining game behaviours as cluster representations in a latent space.<n>The second aggregates over these micro patterns to discover play styles.
  arXiv  Detail & Related papers  (2025-05-01T05:51:19Z)
• Probing Visual Language Priors in VLMs [51.016683265437536]
  We introduce ViLP, a benchmark featuring deliberately out-of-distribution images.<n>Each question in ViLP is coupled with three potential answers and three corresponding images.<n>We propose a self-improving framework in which models generate new VQA data, then apply pixel-level and semantic corruptions to form "good-bad" image pairs for self-training.
  arXiv  Detail & Related papers  (2024-12-31T17:54:29Z)
• Predicting Outcomes in Video Games with Long Short Term Memory Networks [0.39723189359605243]
  Our work attempts to enhance audience engagement within video game tournaments by introducing a real-time method of predicting wins. As a proof of concept, we evaluate our model's performance within a classic, two-player arcade game, Super Street Fighter II Turbo.
  arXiv  Detail & Related papers  (2024-02-24T22:36:23Z)
• MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound [90.1857707251566]
  We introduce MERLOT Reserve, a model that represents videos jointly over time. We replace snippets of text and audio with a MASK token; the model learns by choosing the correct masked-out snippet. Our objective learns faster than alternatives, and performs well at scale.
  arXiv  Detail & Related papers  (2022-01-07T19:00:21Z)
• Predicting Events in MOBA Games: Dataset, Attribution, and Evaluation [37.16502752193698]
  In this work, we collect and release a large-scale dataset containing rich in-game features for the popular MOBA game Honor of Kings. We then propose to predict four types of important events in an interpretable way by attributing the predictions to the input features using two gradient-based attribution methods.
  arXiv  Detail & Related papers  (2020-12-17T07:28:35Z)
• Interpretable Real-Time Win Prediction for Honor of Kings, a Popular Mobile MOBA Esport [51.20042288437171]
  We propose a Two-Stage Spatial-Temporal Network (TSSTN) that can provide accurate real-time win predictions. Experiment results and applications in real-world live streaming scenarios showed that the proposed TSSTN model is effective both in prediction accuracy and interpretability.
  arXiv  Detail & Related papers  (2020-08-14T12:00:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.