Related papers: RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models

URL: http://arxiv.org/abs/2606.18950v2
Date: Thu, 18 Jun 2026 08:25:27 GMT
Title: RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models
Abstract summary: Real-time strategy games can be a natural testbed for Modern Vision-Language Models (VLMs)<n>We present RTSGameBench, which is built on Beyond All Reason, a large-scale RTS game with an expanded battlefield.<n>For VLMs to operate in large-scale RTS games, we provide RTSGameAgent that manages units by an FSM with agentic memory.
Score: 22.162253325526127
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Modern Vision-Language Models (VLMs) often struggle with strategic reasoning, i.e., anticipating and influencing other agents' actions, under uncertainty in competitive and cooperative settings. Real-time strategy (RTS) games can be a natural testbed for diagnosing this limitation, as they demand coordination with allies, adaptation to opponents' strategy, and long-horizon planning under partial observability. However, existing RTS benchmarks offer limited evaluation scope, lack systematic competency diagnosis, and remain fixed in the pre-designed scenario coverage. To address these limitations, we present RTSGameBench, which is built on Beyond All Reason, a large-scale RTS game with an expanded battlefield that demands broader strategy diversity than the existing testbeds. The proposed benchmark provides evaluations through diverse gameplay across various matchup structures, diagnostic assessment via mini-games, each targeting an individual strategic competency, and extensible coverage via a self-evolving generation framework that converts free-form queries into new mini-games, improving over successive cycles. Additionally, for VLMs to operate in large-scale RTS games, we provide RTSGameAgent that manages units by an FSM with agentic memory. We empirically validate that multiple state-of-the-art VLMs do not perform well when matchups demand tighter coordination, multiagent coordination and when task scale increases.

Related papers

Stateful Cooperative Agents Safeguarding LLMs Against Evolving Multi-Turn Attacks [56.45762972211923]
We present a proactive defense framework for securing LLMs against evolving multi-turn adversarial attacks.<n>It employs a cooperative multi-agent architecture in which specialized agents execute complementary defense strategies.<n>We present the EMRA dataset designed to simulate evolving strategies across multi-turn attacks.
arXiv Detail & Related papers (2026-07-31T14:04:49Z)
Spatial Reasoning in LLM Game Agents: Impact of Causal Context and Multi-Step Planning [56.67877550139192]
We conduct experiments across varying model scales, reasoning modes, and planning horizons.<n>We introduce a focused GVGAI benchmark consisting of three custom games with five difficulty levels to isolate spatial navigation.<n>Our results show that while larger models with an enabled thinking mode identify their positions more accurately, overall performance in coordinate matching remains limited for smaller models.
arXiv Detail & Related papers (2026-07-22T12:10:45Z)
MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
arXiv Detail & Related papers (2026-05-28T07:33:47Z)
Odysseus: Scaling VLMs to 100+ Turn Decision-Making in Games via Reinforcement Learning [50.464623632604976]
In this work, we study RL-based training of vision-language models (VLMs) for long-horizon decision-making in Super Mario Land.<n>We propose an adapted variant of PPO with a lightweight turn-level critic, which substantially improves training stability and sample efficiency.<n>We introduce Odysseus, an open training framework for VLM agents, achieving substantial gains across multiple levels of the game.
arXiv Detail & Related papers (2026-05-01T02:05:56Z)
TowerMind: A Tower Defence Game Learning Environment and Benchmark for LLM as Agents [5.173133826653683]
We present TowerMind, a novel environment grounded in the tower defense subgenre of RTS games.<n>We design five benchmark levels to evaluate several widely used Large Language Models.<n>Results reveal a clear performance gap between LLMs and human experts across both capability and hallucination dimensions.
arXiv Detail & Related papers (2026-01-09T16:18:08Z)
LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition [104.81487689011341]
We introduce LM Fight Arena, a novel framework that evaluates large multimodal models in Mortal Kombat II.<n>Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities.
arXiv Detail & Related papers (2025-10-10T02:19:21Z)
PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments [48.892997022500765]
We introduce PillagerBench, a framework for evaluating multi-agent systems in real-time competitive team-vs-team scenarios in Minecraft.<n>We also propose TactiCrafter, an LLM-based multi-agent system that facilitates teamwork through human-readable tactics.<n>Our evaluation demonstrates that TactiCrafter outperforms baseline approaches and showcases adaptive learning through self-play.
arXiv Detail & Related papers (2025-09-07T22:51:12Z)
Society of Mind Meets Real-Time Strategy: A Hierarchical Multi-Agent Framework for Strategic Reasoning [16.35236123729838]
We propose a hierarchical multi-agent framework that employs specialized imitation learning agents under a meta-controller called Strategic Planner (SP)<n>By expert demonstrations, each specialized agent learns a distinctive strategy, such as aerial support or defensive maneuvers, and produces coherent, structured multistep action sequences.<n>The SP then orchestrates these proposals into a single, environmentally adaptive plan that ensures local decisions align with long-term strategies.
arXiv Detail & Related papers (2025-08-08T05:57:12Z)
GameBench: Evaluating Strategic Reasoning Abilities of LLM Agents [4.209869303518743]
We introduce GameBench, a cross-domain benchmark for evaluating strategic reasoning abilities of large language models. Our evaluations use GPT-3 and GPT-4 in their base form along with two scaffolding frameworks designed to enhance strategic reasoning ability: Chain-of-Thought (CoT) prompting and Reasoning Via Planning (RAP) Our results show that none of the tested models match human performance, and at worst GPT-4 performs worse than random action.
arXiv Detail & Related papers (2024-06-07T00:28:43Z)
Conditional Imitation Learning for Multi-Agent Games [89.897635970366]
We study the problem of conditional multi-agent imitation learning, where we have access to joint trajectory demonstrations at training time. We propose a novel approach to address the difficulties of scalability and data scarcity. Our model learns a low-rank subspace over ego and partner agent strategies, then infers and adapts to a new partner strategy by interpolating in the subspace.
arXiv Detail & Related papers (2022-01-05T04:40:13Z)
The Design Of "Stratega": A General Strategy Games Framework [62.997667081978825]
Stratega is a framework for creating turn-based and real-time strategy games. The framework has been built with a focus on statistical forward planning (SFP) agents. We hope that the development of this framework and its respective agents helps to better understand the complex decision-making process in strategy games.
arXiv Detail & Related papers (2020-09-11T20:02:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.