Interpreting learned search: finding a transition model and value function in an RNN that plays Sokoban

• URL: http://arxiv.org/abs/2506.10138v1
• Date: Wed, 11 Jun 2025 19:36:17 GMT
• Title: Interpreting learned search: finding a transition model and value function in an RNN that plays Sokoban
• Authors: Mohammad Taufeeque, Aaron David Tucker, Adam Gleave, Adrià Garriga-Alonso,
• Abstract summary: We partially reverse-engineer a convolutional recurrent neural network (RNN) trained to play the puzzle game Sokoban.<n>Prior work found that this network solves more levels with more test-time compute.
• Score: 3.274397973865673
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We partially reverse-engineer a convolutional recurrent neural network (RNN) trained to play the puzzle game Sokoban with model-free reinforcement learning. Prior work found that this network solves more levels with more test-time compute. Our analysis reveals several mechanisms analogous to components of classic bidirectional search. For each square, the RNN represents its plan in the activations of channels associated with specific directions. These state-action activations are analogous to a value function - their magnitudes determine when to backtrack and which plan branch survives pruning. Specialized kernels extend these activations (containing plan and value) forward and backward to create paths, forming a transition model. The algorithm is also unlike classical search in some ways. State representation is not unified; instead, the network considers each box separately. Each layer has its own plan representation and value function, increasing search depth. Far from being inscrutable, the mechanisms leveraging test-time compute learned in this network by model-free training can be understood in familiar terms.

Related papers

• Disentangling Rich Dynamics from Feature Learning: A Framework for Independent Measurements [5.369150515904139]
  We introduce (1) a measure that quantifies the rich regime independently of performance, and (2) interpretable feature metrics for visualization.<n>We reveal how batch normalization and training set size influence lazy/rich dynamics for VGG16 and ResNet18 on CIFAR-10/100.
  arXiv  Detail & Related papers  (2024-10-05T18:53:48Z)
• Planning in a recurrent neural network that plays Sokoban [6.059513516334741]
  We analyze a recurrent neural network (RNN) trained on Sokoban, a puzzle requiring sequential, irreversible decisions.<n>We find that the RNN has a causal plan representation which predicts its future actions about 50 steps in advance.<n>We extend the trained RNN to significantly larger, out-of-distribution Sokoban puzzles, demonstrating robust representations beyond the training regime.
  arXiv  Detail & Related papers  (2024-07-22T06:57:34Z)
• Generative Kaleidoscopic Networks [2.321684718906739]
  We utilize this property of neural networks to design a dataset kaleidoscope, termed as Generative Kaleidoscopic Networks' We observed this phenomenon to various degrees for the other deep learning architectures like CNNs, Transformers & U-Nets.
  arXiv  Detail & Related papers  (2024-02-19T02:48:40Z)
• Provable Multi-Task Representation Learning by Two-Layer ReLU Neural Networks [69.38572074372392]
  We present the first results proving that feature learning occurs during training with a nonlinear model on multiple tasks. Our key insight is that multi-task pretraining induces a pseudo-contrastive loss that favors representations that align points that typically have the same label across tasks.
  arXiv  Detail & Related papers  (2023-07-13T16:39:08Z)
• How neural networks learn to classify chaotic time series [77.34726150561087]
  We study the inner workings of neural networks trained to classify regular-versus-chaotic time series. We find that the relation between input periodicity and activation periodicity is key for the performance of LKCNN models.
  arXiv  Detail & Related papers  (2023-06-04T08:53:27Z)
• Do deep neural networks have an inbuilt Occam's razor? [1.1470070927586016]
  We show that structured data combined with an intrinsic Occam's razor-like inductive bias towards simple functions counteracts the exponential growth of functions with complexity. This analysis reveals that structured data, combined with an intrinsic Occam's razor-like inductive bias towards (Kolmogorov) simple functions that is strong enough to counteract the exponential growth of functions with complexity, is a key to the success of DNNs.
  arXiv  Detail & Related papers  (2023-04-13T16:58:21Z)
• Discovering dynamical features of Hodgkin-Huxley-type model of physiological neuron using artificial neural network [0.0]
  We consider Hodgkin-Huxley-type system with two fast and one slow variables. For these two systems we create artificial neural networks that are able to reproduce their dynamics. For the bistable model it means that the network being trained only on one brunch of the solutions recovers another without seeing it during the training.
  arXiv  Detail & Related papers  (2022-03-26T19:04:19Z)
• Neural Capacitance: A New Perspective of Neural Network Selection via Edge Dynamics [85.31710759801705]
  Current practice requires expensive computational costs in model training for performance prediction. We propose a novel framework for neural network selection by analyzing the governing dynamics over synaptic connections (edges) during training. Our framework is built on the fact that back-propagation during neural network training is equivalent to the dynamical evolution of synaptic connections.
  arXiv  Detail & Related papers  (2022-01-11T20:53:15Z)
• Why Lottery Ticket Wins? A Theoretical Perspective of Sample Complexity on Pruned Neural Networks [79.74580058178594]
  We analyze the performance of training a pruned neural network by analyzing the geometric structure of the objective function. We show that the convex region near a desirable model with guaranteed generalization enlarges as the neural network model is pruned.
  arXiv  Detail & Related papers  (2021-10-12T01:11:07Z)
• Spatio-Temporal Inception Graph Convolutional Networks for Skeleton-Based Action Recognition [126.51241919472356]
  We design a simple and highly modularized graph convolutional network architecture for skeleton-based action recognition. Our network is constructed by repeating a building block that aggregates multi-granularity information from both the spatial and temporal paths.
  arXiv  Detail & Related papers  (2020-11-26T14:43:04Z)
• Incremental Training of a Recurrent Neural Network Exploiting a Multi-Scale Dynamic Memory [79.42778415729475]
  We propose a novel incrementally trained recurrent architecture targeting explicitly multi-scale learning. We show how to extend the architecture of a simple RNN by separating its hidden state into different modules. We discuss a training algorithm where new modules are iteratively added to the model to learn progressively longer dependencies.
  arXiv  Detail & Related papers  (2020-06-29T08:35:49Z)
• Reusing Trained Layers of Convolutional Neural Networks to Shorten Hyperparameters Tuning Time [1.160208922584163]
  This paper describes a proposal to reuse the weights of hidden (convolutional) layers among different trainings to shorten this process. The experiments compare the training time and the validation loss when reusing and not reusing convolutional layers. They confirm that this strategy reduces the training time while it even increases the accuracy of the resulting neural network.
  arXiv  Detail & Related papers  (2020-06-16T11:39:39Z)
• Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
  This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs. Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-04-06T15:51:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.