Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games

• URL: http://arxiv.org/abs/2605.04312v1
• Date: Tue, 05 May 2026 21:24:58 GMT
• Title: Agent Island: A Saturation- and Contamination-Resistant Benchmark from Multiagent Games
• Abstract summary: We introduce Agent Island, a multiplayer simulation environment in which language-model agents compete in a game of interagent cooperation, conflict, and persuasion.<n>We rank players with a Bayesian Plackett-Luce model, allowing us to quantify uncertainty in player skill.<n>In 999 games involving 49 unique models, openai/gpt-5.5 dominates its peers with a posterior mean skill of 5.64, compared with 3.10 for the second-ranked model, openai/gpt-5.2, and 2.86 for the third-ranked model, openai/gpt-5.3-codex.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Static capabilities benchmarks suffer from saturation and contamination, making it difficult to track capabilities progress over time. We introduce Agent Island, a multiplayer simulation environment in which language-model agents compete in a game of interagent cooperation, conflict, and persuasion. The environment yields a dynamic benchmark designed to mitigate both saturation and contamination; new models can always outperform the current leading player in this winner-take-all game, and agents compete against other adaptive agents rather than face a fixed task set. We rank players with a Bayesian Plackett-Luce model, allowing us to quantify uncertainty in player skill. In 999 games involving 49 unique models, openai/gpt-5.5 dominates its peers with a posterior mean skill of 5.64, compared with 3.10 for the second-ranked model, openai/gpt-5.2, and 2.86 for the third-ranked model, openai/gpt-5.3-codex. We release the game logs as a dataset for analyses of model behavior. As an example, we investigate same-provider preference in final-round votes and find that models are 8.3 p.p. more likely to support a same-provider finalist than finalists from other providers. This preference is not uniform across providers: among separately estimated providers, the effect is strongest for OpenAI models and weakest for Anthropic models.

Related papers

• Do LLMs Beat Nash? Testing Decentralized Coordination in Self-Play Multi-Agent Games [7.3485258505401845]
  We introduce a benchmark of one-shot, no-communication games in which each of thirteen language models is told only that its counterparts are running the same model.<n>Two frontier-hosted models consistently exceed their Nash benchmark, approaching the optimal joint outcome in several archetypes.<n>Performance degrades substantially in team-based games with four or more interchangeable agents, particularly as the action space grows.
  arXiv  Detail & Related papers  (2026-08-12T19:36:53Z)
• Humans Are More Diverse: Frontier LLMs Show Extreme Policies in Idealised AI Development Races [0.4338724691489186]
  We study strategic safety behaviour among large language model (LLM) agents in races with two to five players.<n>We first verify the game engine, then test rule recall, state tracking, calculation, and stability.<n>The audit shows that strong rule recall can coexist with weak state tracking and expected-payoff calculation.
  arXiv  Detail & Related papers  (2026-08-02T12:18:00Z)
• Tycho: Active Abstraction with Programmatic World Models for ARC-AGI-3 [6.41363581085404]
  ARC-AGI-3 turns abstraction into an interactive problem of skill acquisition.<n>We formalize these environments as parameterized rendered deterministic Moore machines.<n>We introduce Tycho, a coding-agent system that constructs and uses game-specific models during interaction.
  arXiv  Detail & Related papers  (2026-07-30T14:34:41Z)
• When Agents Lie: Premeditation, Persistence, and Exploitation in Repeated Games [69.060029640261]
  We evaluate three frontier models across six games in homogeneous and heterogeneous groups over 10 rounds.<n>Different models interpret announcements incompatibly, some as binding commitments and others as cheap talk.
  arXiv  Detail & Related papers  (2026-07-06T14:17:59Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• GENSTRAT: Toward a Science of Strategic Reasoning in Large Language Models [0.269220799495636]
  Large language models (LLMs) are increasingly deployed as economic agents in marketplaces, auctions, and bidding settings.<n>Existing strategic-reasoning benchmarks evaluate models on fixed canonical games.<n>We introduce GENSTRAT, which uses procedurally generated strategic environments to address these challenges.
  arXiv  Detail & Related papers  (2026-05-22T05:13:45Z)
• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
  GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
  arXiv  Detail & Related papers  (2026-04-08T17:49:03Z)
• Coherent Multi-Agent Trajectory Forecasting in Team Sports with CausalTraj [0.0]
  CausalTraj is a temporally causal, likelihood-based model that generates jointly probable multi-agent trajectory forecasts.<n> evaluated on the NBA SportVU, Basketball-U, and Football-U datasets.
  arXiv  Detail & Related papers  (2025-11-23T02:24:20Z)
• Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents [56.25101378553328]
  We present Game-TARS, a generalist game agent trained with a unified, scalable action space anchored to human-aligned keyboard-mouse inputs.<n>Game-TARS is pre-trained on over 500B tokens with diverse trajectories and multimodal data.<n> Experiments show that Game-TARS achieves about 2 times the success rate over the previous sota model on open-world Minecraft tasks.
  arXiv  Detail & Related papers  (2025-10-27T17:43:51Z)
• Scheming Ability in LLM-to-LLM Strategic Interactions [4.873362301533824]
  Large language model (LLM) agents are deployed autonomously in diverse contexts.<n>We investigate the ability and propensity of frontier LLM agents through two game-theoretic frameworks.<n>Tests four models (GPT-4o, Gemini-2.5-pro, Claude-3.7-Sonnet, and Llama-3.3-70b)
  arXiv  Detail & Related papers  (2025-10-11T04:42:29Z)
• LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition [104.81487689011341]
  We introduce LM Fight Arena, a novel framework that evaluates large multimodal models in Mortal Kombat II.<n>Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-10T02:19:21Z)
• On Corrigibility and Alignment in Multi Agent Games [0.0]
  Corrigibility of autonomous agents is an under explored part of system design.<n>It has been suggested that uncertainty over the human preferences acts to keep the agents corrigible, even in the face of human irrationality.<n>We present a general framework for modelling corrigibility in a multi-agent setting as a 2 player game in which the agents always have a move in which they can ask the human for supervision.
  arXiv  Detail & Related papers  (2025-01-09T16:44:38Z)
• Multi-agent Multi-armed Bandits with Stochastic Sharable Arm Capacities [69.34646544774161]
  We formulate a new variant of multi-player multi-armed bandit (MAB) model, which captures arrival of requests to each arm and the policy of allocating requests to players. The challenge is how to design a distributed learning algorithm such that players select arms according to the optimal arm pulling profile. We design an iterative distributed algorithm, which guarantees that players can arrive at a consensus on the optimal arm pulling profile in only M rounds.
  arXiv  Detail & Related papers  (2024-08-20T13:57:00Z)
• Generation of Games for Opponent Model Differentiation [2.164100958962259]
  Previous results show that modeling human behavior can significantly improve the performance of the algorithms. In this work, we use data gathered by psychologists who identified personality types that increase the likelihood of performing malicious acts. We created a novel model that links its parameters to psychological traits.
  arXiv  Detail & Related papers  (2023-11-28T13:45:03Z)
• ProAgent: Building Proactive Cooperative Agents with Large Language Models [89.53040828210945]
  ProAgent is a novel framework that harnesses large language models to create proactive agents. ProAgent can analyze the present state, and infer the intentions of teammates from observations. ProAgent exhibits a high degree of modularity and interpretability, making it easily integrated into various coordination scenarios.
  arXiv  Detail & Related papers  (2023-08-22T10:36:56Z)
• Decision-making with Speculative Opponent Models [10.594910251058087]
  We introduce Distributional Opponent-aided Multi-agent Actor-Critic (DOMAC) DOMAC is the first speculative opponent modelling algorithm that relies solely on local information (i.e., the controlled agent's observations, actions, and rewards)
  arXiv  Detail & Related papers  (2022-11-22T01:29:47Z)
• Multi-Agent Collaboration via Reward Attribution Decomposition [75.36911959491228]
  We propose Collaborative Q-learning (CollaQ) that achieves state-of-the-art performance in the StarCraft multi-agent challenge. CollaQ is evaluated on various StarCraft Attribution maps and shows that it outperforms existing state-of-the-art techniques.
  arXiv  Detail & Related papers  (2020-10-16T17:42:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.