The PokeAgent Challenge: Competitive and Long-Context Learning at Scale

• URL: http://arxiv.org/abs/2603.15563v2
• Date: Tue, 17 Mar 2026 06:05:02 GMT
• Title: The PokeAgent Challenge: Competitive and Long-Context Learning at Scale
• Abstract summary: PokeAgent Challenge is a large-scale benchmark for decisionmaking research.<n>It is built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment.<n>Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon.
• Score: 45.224407977351824
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We present the PokeAgent Challenge, a large-scale benchmark for decision-making research built on Pokemon's multi-agent battle system and expansive role-playing game (RPG) environment. Partial observability, game-theoretic reasoning, and long-horizon planning remain open problems for frontier AI, yet few benchmarks stress all three simultaneously under realistic conditions. PokeAgent targets these limitations at scale through two complementary tracks: our Battling Track, which calls for strategic reasoning and generalization under partial observability in competitive Pokemon battles, and our Speedrunning Track, which requires long-horizon planning and sequential decision-making in the Pokemon RPG. Our Battling Track supplies a dataset of 20M+ battle trajectories alongside a suite of heuristic, RL, and LLM-based baselines capable of high-level competitive play. Our Speedrunning Track provides the first standardized evaluation framework for RPG speedrunning, including an open-source multi-agent orchestration system for modular, reproducible comparisons of harness-based LLM approaches. Our NeurIPS 2025 competition validates both the quality of our resources and the research community's interest in Pokemon, with over 100 teams competing across both tracks and winning solutions detailed in our paper. Participant submissions and our baselines reveal considerable gaps between generalist (LLM), specialist (RL), and elite human performance. Analysis against the BenchPress evaluation matrix shows that Pokemon battling is nearly orthogonal to standard LLM benchmarks, measuring capabilities not captured by existing suites and positioning Pokemon as an unsolved benchmark that can drive RL and LLM research forward. We transition to a living benchmark with a live leaderboard for Battling and self-contained evaluation for Speedrunning at https://pokeagentchallenge.com.

Related papers

• PTCG-Bench: Can LLM Agents Master Pokémon Trading Card Game? [14.698359286590408]
  We present PTCG-Bench, a benchmark built on the Pok'emon Trading Card Game (PTCG)<n>Our experiments show that, although LLM agents can achieve non-trivial gameplay performance, sustained and stable self-evolution remains challenging.<n>We hope that PTCG-Bench will facilitate future research on harness-aware and self-evolving agents in realistic interactive environments.
  arXiv  Detail & Related papers  (2026-05-28T09:16:22Z)
• MINDGAMES: A Live Arena for Evaluating Social and Strategic Reasoning in Multi-Agent LLMs [54.81359054218573]
  We introduce Mindgames, a multi-game arena and evaluation platform for large language models (LLMs)<n>Mindgames provides a unified interaction interface, TrueSkill-based rating, and full trajectory logging across four game environments.<n>We analyze 29,571 multi-agent games with turn-level observations, actions, and rewards, together with MG-Ref, a deterministic offline tournament protocol.
  arXiv  Detail & Related papers  (2026-05-28T07:33:47Z)
• PokeRL: Reinforcement Learning for Pokemon Red [0.0]
  We present PokeRL, a modular system that trains deep reinforcement learning agents to complete early game tasks in Pokemon Red.<n>Our main contributions are a loop-aware environment wrapper around the PyBoy emulator with map masking, a multi-layer anti-loop and anti-spam mechanism, and a dense hierarchical reward design.
  arXiv  Detail & Related papers  (2026-04-12T20:46:47Z)
• GameWorld: Towards Standardized and Verifiable Evaluation of Multimodal Game Agents [76.60994803070436]
  GameWorld is a benchmark for evaluation of Multimodal Large Language Model (MLLM) game agents in browser environments.<n>Two game agent interfaces are studied: (i) computer-use agents that directly emit keyboard and mouse controls, and (ii) generalist multimodal agents that act in a semantic action space.<n>Results across 18 model-interface pairs suggest that even the best performing agent is far from achieving human capabilities on video games.
  arXiv  Detail & Related papers  (2026-04-08T17:49:03Z)
• Large Language Models as Pokémon Battle Agents: Strategic Play and Content Generation [4.782714372521615]
  Pokémon battles demand reasoning about type matchups, statistical trade-offs, and risk assessment.<n>This work examines whether Large Language Models (LLMs) can serve as competent battle agents.<n>We developed a turn-based Pokémon battle system where LLMs select moves based on battle state rather than pre-programmed logic.
  arXiv  Detail & Related papers  (2025-12-19T07:46:29Z)
• LM Fight Arena: Benchmarking Large Multimodal Models via Game Competition [104.81487689011341]
  We introduce LM Fight Arena, a novel framework that evaluates large multimodal models in Mortal Kombat II.<n>Unlike static evaluations, LM Fight Arena provides a fully automated, reproducible, and objective assessment of an LMM's strategic reasoning capabilities.
  arXiv  Detail & Related papers  (2025-10-10T02:19:21Z)
• PillagerBench: Benchmarking LLM-Based Agents in Competitive Minecraft Team Environments [48.892997022500765]
  We introduce PillagerBench, a framework for evaluating multi-agent systems in real-time competitive team-vs-team scenarios in Minecraft.<n>We also propose TactiCrafter, an LLM-based multi-agent system that facilitates teamwork through human-readable tactics.<n>Our evaluation demonstrates that TactiCrafter outperforms baseline approaches and showcases adaptive learning through self-play.
  arXiv  Detail & Related papers  (2025-09-07T22:51:12Z)
• Who is a Better Player: LLM against LLM [53.46608216197315]
  We propose an adversarial benchmarking framework to assess the comprehensive performance of Large Language Models (LLMs) through board games competition.<n>We introduce Qi Town, a specialized evaluation platform that supports 5 widely played games and involves 20 LLM-driven players.
  arXiv  Detail & Related papers  (2025-08-05T06:41:47Z)
• PokéAI: A Goal-Generating, Battle-Optimizing Multi-agent System for Pokemon Red [4.558478169296784]
  We introduce Pok'eAI, the first text-based, multi-agent large language model (LLM) framework designed to autonomously play and progress through Pok'emon Red.<n>Our system consists of three specialized agents-Planning, Execution, and Critique-each with its own memory bank, role, and skill set.
  arXiv  Detail & Related papers  (2025-06-30T10:09:13Z)
• Pokemon Red via Reinforcement Learning [3.548348926427221]
  Pok'emon Red, a classic Game Boy JRPG, presents significant challenges as a testbed for agents.<n>We introduce a simplistic environment and a Deep Reinforcement Learning training methodology, demonstrating a baseline agent that completes an initial segment of the game up to completing Cerulean City.<n>Our experiments include various ablations that reveal vulnerabilities in reward shaping, where agents exploit specific reward signals.
  arXiv  Detail & Related papers  (2025-02-27T09:42:23Z)
• FightLadder: A Benchmark for Competitive Multi-Agent Reinforcement Learning [25.857375787748715]
  We present FightLadder, a real-time fighting game platform, to empower competitive MARL research. We provide implementations of state-of-the-art MARL algorithms for competitive games, as well as a set of evaluation metrics. We demonstrate the feasibility of this platform by training a general agent that consistently defeats 12 built-in characters in single-player mode.
  arXiv  Detail & Related papers  (2024-06-04T08:04:23Z)
• SmartPlay: A Benchmark for LLMs as Intelligent Agents [45.76707302899935]
  SmartPlay consists of 6 different games, including Rock-Paper-Scissors, Tower of Hanoi, Minecraft. Each game challenges a subset of 9 important capabilities of an intelligent LLM agent. Tests include reasoning with object dependencies, planning ahead, spatial reasoning, learning from history, and understanding randomness.
  arXiv  Detail & Related papers  (2023-10-02T18:52:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.