Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences

• URL: http://arxiv.org/abs/2605.30100v1
• Date: Thu, 28 May 2026 15:43:31 GMT
• Title: Chess-World-Model: A 10M-Game Benchmark for Exact State Tracking from Chess Move Sequences
• Abstract summary: We introduce Chess-World-Model, a large-scale state-tracking benchmark built from 10 million real chess games.<n>We benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues.<n>Real-game performance saturates above 18 million parameters, but the random-uniform split remains discriminative up to 40 million, exposing failures otherwise hidden by scale.
• Score: 2.7214777196418645
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: World models require state tracking, which is the ability to maintain a correct latent state across action sequences. Existing benchmarks are often synthetic or language-based, limiting their value as tests of structured state updates in realistic domains. We introduce Chess-World-Model, a large-scale state-tracking benchmark built from 10 million real chess games, where models predict the exact board state reached after a sequence of legal moves. Alongside a held-out real-game split, we include an out-of-distribution split from uniformly random legal play, which tests whether models learn the transition rules rather than shortcuts from common human positions. Prior theoretical and empirical work has shown that Transformers struggle to state-track, while input-dependent linear RNNs require expressive state-transition matrices to do so. We therefore benchmark a causal Transformer, block-diagonal SLiCE, Mamba-3, and Gated DeltaNet with negative eigenvalues under a matched interface and training protocol. The recurrent models strongly outperform the Transformer at 3 and 8 million parameters. Real-game performance saturates above 18 million parameters, but the random-uniform split remains discriminative up to 40 million, exposing failures otherwise hidden by scale. Additionally, ablations show that less expressive state-transition mechanisms reduce performance on the out-of-distribution split for all three recurrent models. Together, these results establish Chess-World-Model as a practical large-scale benchmark for state tracking that exposes failures model scale would otherwise conceal.

Related papers

• The Quantum Shortcut: Complex Phase-State Dynamics Reduce the Optimization Steps of Sequence Models [0.0]
  Sequence models are distinguished by their backbone, the mechanism that routes information across positions.<n>This paper varies a choice that is prior to the backbone and shared by nearly all current models.<n>We study a complex-valued alternative drawn from the mathematics of quantum theory, in which information is carried by the phases of the state.
  arXiv  Detail & Related papers  (2026-08-07T15:12:36Z)
• MASS: Multiplayer World Models with Authoritative Shared State [65.85526248184442]
  MASS (Multiplayer world models with Authoritative Shared State) is inspired by multiplayer game architectures.<n>A learned Logic Engine advances a global, authoritative typed state from joint actions.<n>A learned Rendering Engine generates independent and consistent views for any requested camera on demand.
  arXiv  Detail & Related papers  (2026-08-06T16:47:24Z)
• Apple-$π$: Benchmarking Thinking with Video Towards Law-Grounded Physical Intelligence [54.58886283991723]
  Apple-PI is the first benchmark that anchors video-model evaluation explicitly in physical laws.<n>Apple-PI comprises three components.<n> Benchmarking 11 models shows that current video models remain far from reliable law-grounded world simulators.
  arXiv  Detail & Related papers  (2026-07-17T18:00:05Z)
• Will Scaling Improve Social Simulation with LLMs? [87.15677770205751]
  Large Language Model (LLM) social simulations are a promising research method, but they are not yet faithful enough to be adopted widely.<n>We use scaling laws to study the relationship between LLMs' compute scale, general capability benchmarks, and the fidelity of social simulation.
  arXiv  Detail & Related papers  (2026-07-02T17:30:38Z)
• Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Games [69.57330692969543]
  We introduce RNG-Bench, a benchmark suite designed to isolate a base model's ability to reconstruct past observations.<n>RNG-Bench includes two complementary games: Matching Pairs, where card identities briefly revealed at specific locations must later be recalled, and 3D Maze, where egocentric views must be integrated into a spatial map.<n>The hardest configurations require contexts of roughly 128K tokens and 350 image inputs per episode, and remain far from saturated by frontier MLLMs.
  arXiv  Detail & Related papers  (2026-06-17T17:59:34Z)
• Generalization or Memorization? Brittleness Testing for Chess-Trained Language Models [0.0]
  KinGPT is a character-level language model trained only on (position, best-move) pairs.<n>We show how LLM-Modulo, a verifier-in-the-loop framework, raises RedPajama 3B's best move accuracy from 1.2% to 21.2%.
  arXiv  Detail & Related papers  (2026-05-17T17:49:07Z)
• ClawForge: Generating Executable Interactive Benchmarks for Command-Line Agents [59.626170560327274]
  We present textbfClawForge, a generator-backed benchmark framework for executable command-line categories under state conflict.<n>We instantiate this framework as the ClawForge-Bench (17 scenarios, 6 ability categories)
  arXiv  Detail & Related papers  (2026-05-13T21:34:08Z)
• LLM CHESS: Benchmarking Reasoning and Instruction-Following in LLMs through Chess [30.797553771114746]
  We introduce LLM CHESS, an evaluation framework designed to probe the generalization of reasoning and instruction-following abilities in large language models (LLMs)<n>We rank over 50 open and closed source models by playing against a random opponent using a range of behavioral metrics, including move quality, move legality, hallucinated actions, and game duration.<n>For a subset of top reasoning models, we derive an Elo estimate by playing against a chess engine with variably configured skill, which allows for comparisons between models in an easily understandable way.
  arXiv  Detail & Related papers  (2025-12-01T18:51:08Z)
• Out-of-distribution Tests Reveal Compositionality in Chess Transformers [6.356179251855671]
  We train a 270M parameter chess Transformer and test it on out-of-distribution scenarios, designed to reveal failures of systematic generalization.<n>Our analysis shows that Transformers exhibit compositional generalization, as evidenced by strong rule extrapolation.<n>In a more challenging test, we evaluate the models on variants including Chess960 - a variant of chess where starting positions of pieces are randomized.
  arXiv  Detail & Related papers  (2025-10-23T17:51:28Z)
• TurnBench-MS: A Benchmark for Evaluating Multi-Turn, Multi-Step Reasoning in Large Language Models [5.6525926183880255]
  We introduce TurnBench, a novel benchmark that evaluates multi-turn, multi-step reasoning through an interactive code-breaking task.<n>In each episode, a model must uncover hidden logical or arithmetic rules by making sequential guesses, receiving structured feedback, and integrating clues across multiple rounds.<n>TurnBench includes two modes: Classic, which tests standard reasoning, and Nightmare, which introduces increased complexity and requires robust inferential chains.
  arXiv  Detail & Related papers  (2025-06-02T05:47:50Z)
• The Mamba in the Llama: Distilling and Accelerating Hybrid Models [76.64055251296548]
  We show how to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources.<n>The resulting hybrid model achieves performance comparable to the original Transformer in chat benchmarks.<n>We also introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models.
  arXiv  Detail & Related papers  (2024-08-27T17:56:11Z)
• StARformer: Transformer with State-Action-Reward Representations [45.62939275764248]
  We propose State-Action-Reward Transformer (StARformer) for reinforcement learning. StARformer explicitly models local causal relations to help improve action prediction in long sequences. Our experiments show that StARformer outperforms the state-of-the-art Transformer-based method on Atari benchmarks.
  arXiv  Detail & Related papers  (2021-10-12T17:55:30Z)
• Long Range Arena: A Benchmark for Efficient Transformers [115.1654897514089]
  Long-rangearena benchmark is a suite of tasks consisting of sequences ranging from $1K$ to $16K$ tokens. We systematically evaluate ten well-established long-range Transformer models on our newly proposed benchmark suite.
  arXiv  Detail & Related papers  (2020-11-08T15:53:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.