Emergent World Beliefs: Exploring Transformers in Stochastic Games

• URL: http://arxiv.org/abs/2512.23722v1
• Date: Thu, 18 Dec 2025 19:36:53 GMT
• Title: Emergent World Beliefs: Exploring Transformers in Stochastic Games
• Authors: Adam Kamel, Tanish Rastogi, Michael Ma, Kailash Ranganathan, Kevin Zhu,
• Abstract summary: Transformer-based large language models (LLMs) have demonstrated strong reasoning abilities across diverse fields.<n>We pretrain a GPT-style model on Poker Hand History data and probe its internal activations.<n>Our results demonstrate that the model learns both deterministic structure, such as hand ranks, and features, such as equity, without explicit instruction.
• Score: 2.497936211748472
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer-based large language models (LLMs) have demonstrated strong reasoning abilities across diverse fields, from solving programming challenges to competing in strategy-intensive games such as chess. Prior work has shown that LLMs can develop emergent world models in games of perfect information, where internal representations correspond to latent states of the environment. In this paper, we extend this line of investigation to domains of incomplete information, focusing on poker as a canonical partially observable Markov decision process (POMDP). We pretrain a GPT-style model on Poker Hand History (PHH) data and probe its internal activations. Our results demonstrate that the model learns both deterministic structure, such as hand ranks, and stochastic features, such as equity, without explicit instruction. Furthermore, by using primarily nonlinear probes, we demonstrated that these representations are decodeable and correlate with theoretical belief states, suggesting that LLMs are learning their own representation of the stochastic environment of Texas Hold'em Poker.

Related papers

• BotzoneBench: Scalable LLM Evaluation via Graded AI Anchors [9.224594551677374]
  Large Language Models (LLMs) are increasingly deployed in interactive environments requiring strategic decision-making.<n>Recent game-based evaluations employ LLM-vs-LLM tournaments that produce relative rankings dependent on transient model pools.<n>Here we show that anchoring LLM evaluation to fixed hierarchies of skill-calibrated game Artificial Intelligence (AI) enables linear-time absolute skill measurement.
  arXiv  Detail & Related papers  (2026-01-22T13:15:08Z)
• VAGEN: Reinforcing World Model Reasoning for Multi-Turn VLM Agents [130.70999337445468]
  Key challenge in training Vision-Language Model (VLM) agents, compared to Language Model (LLM) agents, is shift from textual states to complex visual observations.<n>We ask: Can VLM agents construct internal world models through explicit visual state reasoning?<n>We architecturally enforce and reward the agent's reasoning process via reinforcement learning (RL)<n>We find that the agent's reasoning into State Estimation and Transition Modeling is critical for success.
  arXiv  Detail & Related papers  (2025-10-19T16:05:07Z)
• FOUNDER: Grounding Foundation Models in World Models for Open-Ended Embodied Decision Making [32.050134958163184]
  Foundation Models (FMs) and World Models (WMs) offer complementary strengths in task generalization at different levels.<n>We propose FOUNDER, a framework that integrates the generalizable knowledge embedded in FMs with the dynamic modeling capabilities of WMs.<n>We learn a mapping function that grounds FM representations in the WM state space, effectively inferring the agent's physical states in the world simulator from external observations.
  arXiv  Detail & Related papers  (2025-07-15T21:49:49Z)
• Play to Generalize: Learning to Reason Through Game Play [18.085614901290427]
  We propose a novel post-training method, Visual Game Learning (ViGaL), where MLLMs develop generalizable reasoning skills through playing arcade-like games.<n>Our findings suggest that multimodal reasoning can emerge from gameplay, pointing to a promising strategy of designing surrogate tasks for RL post-training.
  arXiv  Detail & Related papers  (2025-06-09T17:59:57Z)
• Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learning [65.85335291827086]
  This paper tries to learn and understand underlying semantic variations from distracting videos via offline-to-online latent distillation and flexible disentanglement constraints.<n>We pretrain the action-free video prediction model offline with disentanglement regularization to extract semantic knowledge from distracting videos.<n>For finetuning in the online environment, we exploit the knowledge from the pretrained model and introduce a disentanglement constraint to the world model.
  arXiv  Detail & Related papers  (2025-03-11T13:50:22Z)
• Transformers Use Causal World Models in Maze-Solving Tasks [49.67445252528868]
  We identify World Models in transformers trained on maze-solving tasks.<n>We find that it is easier to activate features than to suppress them.<n> positional encoding schemes appear to influence how World Models are structured within the model's residual stream.
  arXiv  Detail & Related papers  (2024-12-16T15:21:04Z)
• Mastering Board Games by External and Internal Planning with Language Models [30.782334791241556]
  We show that search-based planning can yield significant improvements in Large Language Models game-playing strength.<n>We introduce, compare and contrast two major approaches: in external search, the model guides Monte Carlo Tree Search rollouts and evaluations without calls to an external game engine, and in internal search, the model is trained to generate in-context a linearized tree of search and a resulting final choice.<n>Our proposed approach, combining search with domain knowledge, is not specific to board games, hinting at more general future applications.
  arXiv  Detail & Related papers  (2024-12-02T18:56:51Z)
• Emergent World Models and Latent Variable Estimation in Chess-Playing Language Models [0.0]
  We train a GPT model on Othello games and find that the model learned an internal representation of the board state. We extend this work into the more complex domain of chess, training on real games and investigating our model's internal representations. Unlike Li et al.'s prior synthetic dataset approach, our analysis finds that the model also learns to estimate latent variables like player skill to better predict the next character.
  arXiv  Detail & Related papers  (2024-03-21T18:53:23Z)
• A Theoretical Analysis of Self-Supervised Learning for Vision Transformers [66.08606211686339]
  Masked autoencoders (MAE) and contrastive learning (CL) capture different types of representations.<n>We study the training dynamics of one-layer softmax-based vision transformers (ViTs) on both MAE and CL objectives.
  arXiv  Detail & Related papers  (2024-03-04T17:24:03Z)
• Explainability for Large Language Models: A Survey [59.67574757137078]
  Large language models (LLMs) have demonstrated impressive capabilities in natural language processing. This paper introduces a taxonomy of explainability techniques and provides a structured overview of methods for explaining Transformer-based language models.
  arXiv  Detail & Related papers  (2023-09-02T22:14:26Z)
• FL Games: A federated learning framework for distribution shifts [71.98708418753786]
  Federated learning aims to train predictive models for data that is distributed across clients, under the orchestration of a server. We propose FL Games, a game-theoretic framework for federated learning for learning causal features that are invariant across clients.
  arXiv  Detail & Related papers  (2022-05-23T07:51:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.