Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments

• URL: http://arxiv.org/abs/2608.09128v1
• Date: Mon, 10 Aug 2026 05:12:56 GMT
• Title: Social Gym and SPaRTan: Benchmarking and Improving LLM Social Reasoning via Multi-Agent Game Tournaments
• Abstract summary: We introduce Social Gym, an environment of 21 multi-agent social games with a cross-game leaderboard.<n>We then propose SPaRTan, a training-free self-improvement loop.<n>Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance.
• Score: 33.6358292917346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM agents are increasingly deployed in multi-agent social settings where they must cooperate, negotiate, and adapt to other agents. Measuring and improving these social skills is hard because, unlike math or logic, social interaction offers no objective ground truth: evaluations fall back on LLM judges, which are costly, subjective, and noisy, and models get no reliable signal to learn from. To address both, we first introduce Social Gym, an environment of 21 multi-agent social games (e.g., Werewolves, Resistance, Spyfall) whose rule-decided outcomes make agent performance verifiable and objective, with an Elo tournament that produces a cross-game leaderboard. Benchmarking experiments show that while GPT-5-mini tops the leaderboard, no model excels at all games uniformly or in all game roles, pointing to limitations of social reasoning. Motivated by this, we additionally propose SPaRTan (Self-Play and Reflect-Transfer), a training-free self-improvement loop: a model plays a game, reflects on its trajectories and their outcomes to produce a transferable playbook, and applies that playbook in subsequent games. Our results show that SPaRTan playbooks help GPT-5-mini agents level their performance on weaker roles, but largely do not improve Qwen3-32B's performance. Together, Social Gym and SPaRTan offer a reproducible, verifiable foundation for measuring and improving LLM social reasoning without weight updates.

Related papers

• Can Agents Deceive? Evaluating Reasoning and Deception in ParliamentBench using a Social Deduction Game [23.840355539083372]
  Large language models (LLMs) are deployed as agents in high-stakes settings, such as medical and legal systems.<n>We evaluate LLMs in scenarios that require deception, persuasion, and reasoning under information asymmetry.<n>We introduce three novel metrics that isolate social deduction, reasoning, and deceptive consistency.
  arXiv  Detail & Related papers  (2026-07-30T12:54:17Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• SocialGrid: A Benchmark for Planning and Social Reasoning in Embodied Multi-Agent Systems [38.033245595823026]
  We introduce SocialGrid, an embodied multi-agent environment that evaluates agents on planning, task execution, and social reasoning.<n>Our evaluations reveal that even the strongest open model achieves below 60% accuracy in task completion and planning.<n>SocialGrid provides automatic failure analysis and fine-grained metrics, enabling developers to diagnose and improve their agents.
  arXiv  Detail & Related papers  (2026-04-17T12:51:46Z)
• Who is a Better Player: LLM against LLM [53.46608216197315]
  We propose an adversarial benchmarking framework to assess the comprehensive performance of Large Language Models (LLMs) through board games competition.<n>We introduce Qi Town, a specialized evaluation platform that supports 5 widely played games and involves 20 LLM-driven players.
  arXiv  Detail & Related papers  (2025-08-05T06:41:47Z)
• lmgame-Bench: How Good are LLMs at Playing Games? [60.01834131847881]
  We study the major challenges in using popular video games to evaluate modern large language model (LLM) agents.<n>We introduce lmgame-Bench to turn games into reliable evaluations.
  arXiv  Detail & Related papers  (2025-05-21T06:02:55Z)
• Toward Optimal LLM Alignments Using Two-Player Games [86.39338084862324]
  In this paper, we investigate alignment through the lens of two-agent games, involving iterative interactions between an adversarial and a defensive agent. We theoretically demonstrate that this iterative reinforcement learning optimization converges to a Nash Equilibrium for the game induced by the agents. Experimental results in safety scenarios demonstrate that learning in such a competitive environment not only fully trains agents but also leads to policies with enhanced generalization capabilities for both adversarial and defensive agents.
  arXiv  Detail & Related papers  (2024-06-16T15:24:50Z)
• LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplay [55.12945794835791]
  Using Avalon as a testbed, we employ system prompts to guide LLM agents in gameplay. We propose a novel framework, tailored for Avalon, features a multi-agent system facilitating efficient communication and interaction. Results affirm the framework's effectiveness in creating adaptive agents and suggest LLM-based agents' potential in navigating dynamic social interactions.
  arXiv  Detail & Related papers  (2023-10-23T14:35:26Z)
• Playing repeated games with Large Language Models [23.57695486747895]
  LLMs are increasingly used in applications where they interact with humans and other agents.<n>We let different LLMs play finitely repeated $2times2$ games with each other, with human-like strategies, and actual human players.<n>Our results show that LLMs perform particularly well at self-interested games like the iterated Prisoner's Dilemma family.<n>However, they behave sub-optimally in games that require coordination, like the Battle of the Sexes.
  arXiv  Detail & Related papers  (2023-05-26T12:17:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.