Explaining Attention with Program Synthesis

• URL: http://arxiv.org/abs/2606.19317v1
• Date: Wed, 17 Jun 2026 17:40:55 GMT
• Title: Explaining Attention with Program Synthesis
• Abstract summary: A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descriptions.<n>We propose an approach for approximating the behavior of components of deep networks with executable programs.
• Score: 45.88028371034407
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A longstanding goal of research on interpretable deep learning is to replace opaque neural computations with human-meaningful symbolic descriptions. In this paper, we propose an approach for approximating the behavior of components of deep networks with executable programs. We focus on attention heads in transformer language models. For a given head, we first compute its associated attention matrices on a collection of randomly selected training examples. Next, we prompt a pre-trained language model with a summary of these matrices, and instruct it to generate a set of Python programs that can reproduce the associated attention patterns given only text from the input sentence. Finally, we re-rank programs according to how well our final set of programs predict behavior on held-out inputs. We demonstrate that a set of fewer than 1,000 such generated programs can reproduce the attention patterns of heads in GPT-2, TinyLlama-1.1B, and Llama-3B, achieving an average Intersection-over-Union similarity above 75% on TinyStories. Moreover, the best-fit programs can replace neural attention heads without substantially affecting model behavior: replacing 25% of attention heads with programmatic surrogates across the three models incurs only a 16% average perplexity increase, while maintaining performance on a variety of downstream question answering benchmarks. This work contributes a scalable pipeline for reverse-engineering attention heads in transformer models using human-readable, executable code, advancing a path toward symbolic transparency in neural models.

Related papers

• Pretraining Language Models to Ponder in Continuous Space [50.52734567589996]
  We introduce this pondering process into language models by repeatedly invoking the forward process within a single token generation step.<n>We show that the model can learn to ponder in this way through self-supervised learning, without any human annotations.
  arXiv  Detail & Related papers  (2025-05-27T03:47:33Z)
• Learning Program Behavioral Models from Synthesized Input-Output Pairs [70.9524884086882]
  We introduce Modelizer, a framework that learns a model from its input/output behavior using neural machine translation algorithms.<n>Modelizer mocks the original program, achieving up to 95.4% accuracy and a BLEU score of 0.98 with standard error 0.04 in mocking real-world applications.<n>We foresee several applications of these models, especially as the output of the program can be any aspect of program behavior.
  arXiv  Detail & Related papers  (2024-07-11T15:25:02Z)
• CodeArt: Better Code Models by Attention Regularization When Symbols Are Lacking [12.458135956476639]
  Transformer based code models have impressive performance in many software engineering tasks. However, their effectiveness degrades when symbols are missing or not informative. We propose a new method to pre-train general code models when symbols are lacking.
  arXiv  Detail & Related papers  (2024-02-19T05:13:22Z)
• In-Context Language Learning: Architectures and Algorithms [73.93205821154605]
  We study ICL through the lens of a new family of model problems we term in context language learning (ICLL) We evaluate a diverse set of neural sequence models on regular ICLL tasks.
  arXiv  Detail & Related papers  (2024-01-23T18:59:21Z)
• Deep Graph Reprogramming [112.34663053130073]
  "Deep graph reprogramming" is a model reusing task tailored for graph neural networks (GNNs) We propose an innovative Data Reprogramming paradigm alongside a Model Reprogramming paradigm.
  arXiv  Detail & Related papers  (2023-04-28T02:04:29Z)
• Masked Autoencoding for Scalable and Generalizable Decision Making [93.84855114717062]
  MaskDP is a simple and scalable self-supervised pretraining method for reinforcement learning and behavioral cloning. We find that a MaskDP model gains the capability of zero-shot transfer to new BC tasks, such as single and multiple goal reaching.
  arXiv  Detail & Related papers  (2022-11-23T07:04:41Z)
• Foundation Posteriors for Approximate Probabilistic Inference [11.64841553345271]
  We formulate inference as masked language modeling in a probabilistic program. We train a neural network to unmask the random values, defining an approximate posterior distribution. We show the efficacy of the approach, zero-shot and fine-tuned, on a benchmark of STAN programs.
  arXiv  Detail & Related papers  (2022-05-19T17:42:37Z)
• Towards Neural Functional Program Evaluation [0.5586191108738562]
  We introduce a new program generation mechanism that allows control over syntactic sugar for semantically equivalent programs. Experiments reveal that neural functional program evaluation performs surprisingly well, achieving high 90% exact program match scores.
  arXiv  Detail & Related papers  (2021-12-09T00:20:29Z)
• On the Generalizability of Neural Program Models with respect to Semantic-Preserving Program Transformations [25.96895574298886]
  We evaluate the generalizability of neural program models with respect to semantic-preserving transformations. We use three Java datasets of different sizes and three state-of-the-art neural network models for code. Our results suggest that neural program models based on data and control dependencies in programs generalize better than neural program models based only on abstract syntax trees.
  arXiv  Detail & Related papers  (2020-07-31T20:39:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.