Transformers Linearly Represent Highly Structured World Models

• URL: http://arxiv.org/abs/2605.18847v1
• Date: Wed, 13 May 2026 07:59:05 GMT
• Title: Transformers Linearly Represent Highly Structured World Models
• Abstract summary: We train an 8-layer transformer on Sudoku solving traces and perform a mechanistic analysis of its internal computation.<n>We identify a naked-single circuit: a small set of dedicated neurons in the final layer, each individually detecting when exactly one digit remains possible for a specific cell.
• Score: 2.4732707667151286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Do transformers, when trained on sequential reasoning traces, build internal models of the underlying task? And if so, does the structure of those internal representations mirror the structure of the domain? We train an 8-layer transformer on Sudoku solving traces and perform a mechanistic analysis of its internal computation. We establish two results. First, the model builds a substructure world model: it does not represent the board state cell by cell, as a human analyst would expect, but organizes information around the rows, columns, and boxes that Sudoku's constraints act on. Second, we identify a naked-single circuit: a small set of dedicated neurons in the final MLP layer, each individually detecting when exactly one digit remains possible for a specific cell, and reliably promoting that digit. These findings show that the geometry of an emergent world model is shaped by the constraint algebra of the domain, not its surface presentation, and that the resulting decision circuit is sparse, monosemantic, and fully interpretable. More broadly, they demonstrate that mechanistic interpretability tools can recover an end-to-end algorithmic account of how a transformer solves a combinatorial reasoning task.

Related papers

• Seeing Through Circuits: Faithful Mechanistic Interpretability for Vision Transformers [54.2202951543048]
  We investigate whether useful mechanistic circuits can be identified through computational graphs in vision transformers.<n>We propose an effective method for Automatic Visual Circuit Discovery (Vi-CD) that recovers class-specific circuits for classification.<n>We find that insightful and actionable edge-based circuits can be recovered from vision transformers, adding transparency to the internal computations of these models.
  arXiv  Detail & Related papers  (2026-04-15T23:25:46Z)
• Transformers learn factored representations [62.86679034549244]
  Transformers pretrained via next token prediction learn to factor their world into parts.<n>We formalize two representational hypotheses: (1) a representation in the product space of all factors, whose dimension grows exponentially with the number of parts.<n>We derive precise predictions about the geometric structure of activations for each, including the number of subspaces, their dimensionality, and the arrangement of context embeddings within them.<n>This provides a principled explanation for why transformers decompose the world into parts, and suggests that interpretable low dimensional structure may persist even in models trained on complex data.
  arXiv  Detail & Related papers  (2026-02-02T17:49:06Z)
• Structure Development in List-Sorting Transformers [0.0]
  We study how a one-layer attention-only transformer develops relevant structures while learning to sort lists of numbers.<n>At the end of training, the model organizes its attention heads in two main modes that we refer to as vocabulary-splitting and copy-suppression.
  arXiv  Detail & Related papers  (2025-01-30T15:56:25Z)
• Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles [14.777731830394357]
  We study if causal language modeling can learn a complex task such as solving Sudoku puzzles. We show that Transformer models trained on this synthetic task can indeed learn to solve Sudokus. We also extend our analysis to Zebra puzzles (known as Einstein puzzles) and show that the model solves $92.04 %$ of the puzzles fully correctly.
  arXiv  Detail & Related papers  (2024-09-16T17:42:15Z)
• Can Transformers Do Enumerative Geometry? [44.99833362998488]
  We introduce a Transformer-based approach to computational enumerative geometry.<n>We compute intersection numbers across a value range from $10-45$ to $1045$.<n>We show that the network is implicitly modeling the Virasoro constraints in a purely data-driven manner.
  arXiv  Detail & Related papers  (2024-08-27T09:44:01Z)
• Structured World Representations in Maze-Solving Transformers [3.75591091941815]
  This work focuses on the abstractions formed by small transformer models. We find evidence for the consistent emergence of structured internal representations of maze topology and valid paths. We also take steps towards deciphering the circuity of path-following by identifying attention heads.
  arXiv  Detail & Related papers  (2023-12-05T08:24:26Z)
• Transformers are uninterpretable with myopic methods: a case study with bounded Dyck grammars [36.780346257061495]
  Interpretability methods aim to understand the algorithm implemented by a trained model. We take a critical view of methods that exclusively focus on individual parts of the model.
  arXiv  Detail & Related papers  (2023-12-03T15:34:46Z)
• Mapping of attention mechanisms to a generalized Potts model [50.91742043564049]
  We show that training a neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method. We also compute the generalization error of self-attention in a model scenario analytically using the replica method.
  arXiv  Detail & Related papers  (2023-04-14T16:32:56Z)
• How Do Transformers Learn Topic Structure: Towards a Mechanistic Understanding [56.222097640468306]
  We provide mechanistic understanding of how transformers learn "semantic structure" We show, through a combination of mathematical analysis and experiments on Wikipedia data, that the embedding layer and the self-attention layer encode the topical structure.
  arXiv  Detail & Related papers  (2023-03-07T21:42:17Z)
• Transformers learn in-context by gradient descent [58.24152335931036]
  Training Transformers on auto-regressive objectives is closely related to gradient-based meta-learning formulations. We show how trained Transformers become mesa-optimizers i.e. learn models by gradient descent in their forward pass.
  arXiv  Detail & Related papers  (2022-12-15T09:21:21Z)
• Characterizing Intrinsic Compositionality in Transformers with Tree Projections [72.45375959893218]
  neural models like transformers can route information arbitrarily between different parts of their input. We show that transformers for three different tasks become more treelike over the course of training. These trees are predictive of model behavior, with more tree-like models generalizing better on tests of compositional generalization.
  arXiv  Detail & Related papers  (2022-11-02T17:10:07Z)
• Transformers Learn Shortcuts to Automata [52.015990420075944]
  We find that a low-depth Transformer can represent the computations of any finite-state automaton. We show that a Transformer with $O(log T)$ layers can exactly replicate the computation of an automaton on an input sequence of length $T$. We further investigate the brittleness of these solutions and propose potential mitigations.
  arXiv  Detail & Related papers  (2022-10-19T17:45:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.