Who is a Better Player: LLM against LLM

• URL: http://arxiv.org/abs/2508.04720v1
• Date: Tue, 05 Aug 2025 06:41:47 GMT
• Title: Who is a Better Player: LLM against LLM
• Authors: Yingjie Zhou, Jiezhang Cao, Farong Wen, Li Xu, Yanwei Jiang, Jun Jia, Ronghui Li, Xiaohong Liu, Yu Zhou, Xiongkuo Min, Jie Guo, Zicheng Zhang, Guangtao Zhai,
• Abstract summary: We propose an adversarial benchmarking framework to assess the comprehensive performance of Large Language Models (LLMs) through board games competition.<n>We introduce Qi Town, a specialized evaluation platform that supports 5 widely played games and involves 20 LLM-driven players.
• Score: 53.46608216197315
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adversarial board games, as a paradigmatic domain of strategic reasoning and intelligence, have long served as both a popular competitive activity and a benchmark for evaluating artificial intelligence (AI) systems. Building on this foundation, we propose an adversarial benchmarking framework to assess the comprehensive performance of Large Language Models (LLMs) through board games competition, compensating the limitation of data dependency of the mainstream Question-and-Answer (Q&A) based benchmark method. We introduce Qi Town, a specialized evaluation platform that supports 5 widely played games and involves 20 LLM-driven players. The platform employs both the Elo rating system and a novel Performance Loop Graph (PLG) to quantitatively evaluate the technical capabilities of LLMs, while also capturing Positive Sentiment Score (PSS) throughout gameplay to assess mental fitness. The evaluation is structured as a round-robin tournament, enabling systematic comparison across players. Experimental results indicate that, despite technical differences, most LLMs remain optimistic about winning and losing, demonstrating greater adaptability to high-stress adversarial environments than humans. On the other hand, the complex relationship between cyclic wins and losses in PLGs exposes the instability of LLMs' skill play during games, warranting further explanation and exploration.

Related papers

• ZeroSumEval: An Extensible Framework For Scaling LLM Evaluation with Inter-Model Competition [14.753916893216129]
  We introduce ZeroSumEval, a dynamic, competition-based, and evolving evaluation framework for Large Language Models (LLMs)<n>ZeroSumEval encompasses a diverse suite of games, including security challenges (Capture the Flag), classic board games (chess), and knowledge tests (MathQuiz)
  arXiv  Detail & Related papers  (2025-03-10T16:54:27Z)
• GAMEBoT: Transparent Assessment of LLM Reasoning in Games [54.49589494014147]
  GAMEBoT is a gaming arena designed for rigorous assessment of Large Language Models.<n>We benchmark 17 prominent LLMs across eight games, encompassing various strategic abilities and game characteristics.<n>Our results suggest that GAMEBoT presents a significant challenge, even when LLMs are provided with detailed CoT prompts.
  arXiv  Detail & Related papers  (2024-12-18T08:32:53Z)
• TMGBench: A Systematic Game Benchmark for Evaluating Strategic Reasoning Abilities of LLMs [45.12542636218608]
  We propose TMGBench, characterized by comprehensive game type coverage, diverse scenarios and flexible game organization.<n>Specifically, we incorporate all 144 game types summarized by the Robinson-Goforth topology of 2x2 games, constructed as classic games in our benchmark.<n>To provide a sustainable evaluation framework adaptable to increasingly powerful LLMs, we treat the aforementioned games as atomic units.
  arXiv  Detail & Related papers  (2024-10-14T13:15:34Z)
• FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning [25.857375787748715]
  We present FightLadder, a real-time fighting game platform, to empower competitive MARL research. We provide implementations of state-of-the-art MARL algorithms for competitive games, as well as a set of evaluation metrics. We demonstrate the feasibility of this platform by training a general agent that consistently defeats 12 built-in characters in single-player mode.
  arXiv  Detail & Related papers  (2024-06-04T08:04:23Z)
• Auto-Arena: Automating LLM Evaluations with Agent Peer Battles and Committee Discussions [77.66677127535222]
  Auto-Arena is an innovative framework that automates the entire evaluation process using LLM-powered agents. In our experiments, Auto-Arena shows a 92.14% correlation with human preferences, surpassing all previous expert-annotated benchmarks.
  arXiv  Detail & Related papers  (2024-05-30T17:19:19Z)
• GTBench: Uncovering the Strategic Reasoning Limitations of LLMs via Game-Theoretic Evaluations [87.99872683336395]
  Large Language Models (LLMs) are integrated into critical real-world applications. This paper evaluates LLMs' reasoning abilities in competitive environments. We first propose GTBench, a language-driven environment composing 10 widely recognized tasks.
  arXiv  Detail & Related papers  (2024-02-19T18:23:36Z)
• Leveraging Word Guessing Games to Assess the Intelligence of Large Language Models [105.39236338147715]
  The paper is inspired by the popular language game Who is Spy'' We develop DEEP to evaluate LLMs' expression and disguising abilities. We then introduce SpyGame, an interactive multi-agent framework.
  arXiv  Detail & Related papers  (2023-10-31T14:37:42Z)
• GameEval: Evaluating LLMs on Conversational Games [93.40433639746331]
  We propose GameEval, a novel approach to evaluating large language models (LLMs) GameEval treats LLMs as game players and assigns them distinct roles with specific goals achieved by launching conversations of various forms. We show that GameEval can effectively differentiate the capabilities of various LLMs, providing a comprehensive assessment of their integrated abilities to solve complex problems.
  arXiv  Detail & Related papers  (2023-08-19T14:33:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.