Darkness Visible: Reading the Exception Handler of a Language Model

• URL: http://arxiv.org/abs/2604.04756v2
• Date: Tue, 07 Apr 2026 14:21:35 GMT
• Title: Darkness Visible: Reading the Exception Handler of a Language Model
• Abstract summary: GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure.<n>This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The final MLP of GPT-2 Small exhibits a fully legible routing program -- 27 named neurons organized into a three-tier exception handler -- while the knowledge it routes remains entangled across ~3,040 residual neurons. We decompose all 3,072 neurons (to numerical precision) into: 5 fused Core neurons that reset vocabulary toward function words, 10 Differentiators that suppress wrong candidates, 5 Specialists that detect structural boundaries, and 7 Consensus neurons that each monitor a distinct linguistic dimension. The consensus-exception crossover -- where MLP intervention shifts from helpful to harmful -- is statistically sharp (bootstrap 95% CIs exclude zero at all consensus levels; crossover between 4/7 and 5/7). Three experiments show that "knowledge neurons" (Dai et al., 2022), at L11 of this model, function as routing infrastructure rather than fact storage: the MLP amplifies or suppresses signals already present in the residual stream from attention, scaling with contextual constraint. A garden-path experiment reveals a reversed garden-path effect -- GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure. This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11. Code and data: https://github.com/pbalogh/transparent-gpt2

Related papers

• NeuronSoup: Evolving Asynchronous, Shared-Neuron Temporal Graphs without Backpropagation [0.8122270502556375]
  We present NeuronSoup, a neural computation architecture that replaces synchronous layer-by-layer processing with asynchronous, delay-mediated signal propagation.<n>The entire architecture is co-evolved by a genetic algorithm operating on a flat real-valued genome of 14,602 genes.<n>On 10-class MNIST digit classification using frozen ResNet18 features as input, the system evolves a network of 204 active paths through 266 hidden neurons.
  arXiv  Detail & Related papers  (2026-07-16T17:18:59Z)
• Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs [9.127363793428119]
  Perturbation probing generates task-specific causal hypotheses for FFN neurons in large language models.<n>Across eight behavioral circuits, 13 models, and four architecture families, we identify two circuit structures that organize LLM behavior.
  arXiv  Detail & Related papers  (2026-04-30T04:13:33Z)
• Fundamental Limits of Neural Network Sparsification: Evidence from Catastrophic Interpretability Collapse [7.167095059974211]
  This work investigates feature survival under severe capacity constraints in Variational Autoencoder--Sparse Autoencoder architectures.<n>We introduce an adaptive sparsity scheduling framework that progressively reduces active neurons from 500 to 50 over 50 training epochs.<n>Our key finding reveals a pervasive paradox: while global representation quality remains stable, local feature interpretability collapses systematically.
  arXiv  Detail & Related papers  (2026-03-18T00:16:38Z)
• The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers [0.0]
  We show that layers in language models perform binary routing of continuous signals.<n>We find that specific neurons implement a consensus architecture that are 93-98% mutually exclusive.<n>We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization.
  arXiv  Detail & Related papers  (2026-03-11T17:14:57Z)
• Language Model Circuits Are Sparse in the Neuron Basis [50.460651620833055]
  We show that textbfMLP neurons are as sparse a feature basis as SAEs.<n>This work advances automated interpretability of language models without additional training costs.
  arXiv  Detail & Related papers  (2026-01-30T05:41:19Z)
• A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy [7.405817106579332]
  Behavioral alignment in large language models is often achieved through broad fine-tuning.<n>We propose a method for alignment that identifies and updates only the neurons most responsible for a given behavior.<n>Our results show that sparse, neuron-level updates offer a scalable and precise alternative to full-model fine-tuning.
  arXiv  Detail & Related papers  (2026-01-26T20:20:13Z)
• Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch Interventions [4.032680910442999]
  Polysemanticity Index (PSI) is a null-calibrated metric that quantifies when a neuron's top activations decompose into semantically distinct clusters.<n>On a pretrained ResNet-50 evaluated with Tiny-ImageNet images, PSI identifies neurons whose activation sets split into coherent, nameable prototypes.
  arXiv  Detail & Related papers  (2025-08-23T08:48:59Z)
• Confidence Regulation Neurons in Language Models [91.90337752432075]
  This study investigates the mechanisms by which large language models represent and regulate uncertainty in next-token predictions. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. token frequency neurons, which we describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution.
  arXiv  Detail & Related papers  (2024-06-24T01:31:03Z)
• Interpreting the Second-Order Effects of Neurons in CLIP [73.54377859089801]
  We interpret the function of individual neurons in CLIP by automatically describing them using text.<n>We present the "second-order lens", analyzing the effect flowing from a neuron through the later attention heads, directly to the output.<n>Our results indicate that an automated interpretation of neurons can be used for model deception and for introducing new model capabilities.
  arXiv  Detail & Related papers  (2024-06-06T17:59:52Z)
• Are Neurons Actually Collapsed? On the Fine-Grained Structure in Neural Representations [28.249932563821712]
  Recent work has observed an intriguing ''Neural Collapse'' phenomenon in well-trained neural networks. This appears to suggest that the last-layer representations are completely determined by the labels. We show that the apparent collapse hides important fine-grained structure in the representations.
  arXiv  Detail & Related papers  (2023-06-29T17:07:34Z)
• Improved Convergence Guarantees for Shallow Neural Networks [91.3755431537592]
  We prove convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances, adversarial labels. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the NTK regime''
  arXiv  Detail & Related papers  (2022-12-05T14:47:52Z)
• Redundant representations help generalization in wide neural networks [71.38860635025907]
  We study the last hidden layer representations of various state-of-the-art convolutional neural networks. We find that if the last hidden representation is wide enough, its neurons tend to split into groups that carry identical information, and differ from each other only by statistically independent noise.
  arXiv  Detail & Related papers  (2021-06-07T10:18:54Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.