Reading Task Failure Off the Activations: A Sparse-Feature Audit of GPT-2 Small on Indirect Object Identification

• URL: http://arxiv.org/abs/2605.22719v1
• Date: Thu, 21 May 2026 16:55:27 GMT
• Title: Reading Task Failure Off the Activations: A Sparse-Feature Audit of GPT-2 Small on Indirect Object Identification
• Abstract summary: We report a small, reproducible audit of sparse-autoencoder (SAE) features of GPT-2 small fire differently on failed versus successful trials.<n>On 300 prompts, GPT-2 small reaches 79.7% accuracy; 146 of the 24,576 features in the layer-8 residual-stream SAE release of Bloom (2024) clear a Holm-corrected significance threshold.<n>The strongest single correlate of failure -- feature 17,491, d=+2.93, Neuronpedia label 'cryptographic keys' -- is essentially silent except when the prompt's transferred object is 'the keys'
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We report a small, reproducible audit of which sparse-autoencoder (SAE) features of GPT-2 small fire differently on failed versus successful trials of the Indirect Object Identification (IOI) task. On 300 prompts, GPT-2 small reaches 79.7% accuracy; 146 of the 24,576 features in the layer-8 residual-stream SAE release of Bloom (2024) clear a Holm-corrected significance threshold and 105 reach a large effect size (|Cohen's d| > 0.8). The strongest single correlate of failure -- feature 17,491, d=+2.93, Neuronpedia label 'cryptographic keys' -- is essentially silent except when the prompt's transferred object is 'the keys,' on which GPT-2 small fails 93.3% of the time vs. 7.5% on the other seven objects (Fisher exact p = 8.79 x 10^-33). We put this correlate through three controls that a mechanistic claim should pass. (i) A causal ablation: zeroing feature 17,491 in the residual stream across all token positions of the 45 keys prompts does not restore accuracy (6.7% -> 4.4%); the feature is a correlate, not a sufficient cause at this layer. (ii) A representation baseline: a logistic regression on the raw 768-dimensional residual stream reaches 5-fold ROC AUC = 0.929, matching the top-100 SAE features (0.927); the SAE basis adds interpretability, not predictive power. (iii) A seed-robustness check: across five random seeds the keys-subset failure rate stays in 75.0--93.3% (the behavioural effect is real), but feature 17,491 is the top-|d| feature in only 1 of 5 runs. The methodological contribution is therefore the audit pipeline (cheap, model-agnostic, surfaces named correlates) rather than any single feature found through it. We release the code, the 300-prompt corpus, the 300x24,576 activation matrix, the ablation and baseline scripts, and the figures. The full pipeline runs on a laptop (Apple M3 Max, no discrete GPU).

Related papers

• Where Decoder Cosine Similarity Fails for SAE Feature Flow Discovery [0.20415910628419062]
  We study how state features and update features interact to produce downstream residual features.<n>Ultimately, our results suggest that feature flow atlases can serve as diagnostics of representation-update mechanisms.
  arXiv  Detail & Related papers  (2026-09-11T08:44:13Z)
• When Graph-JEPA Learns the Wrong Thing: Diagnosing and Repairing Category-Conditional Collapse [3.0255780640228984]
  Joint-embedding predictive architectures are selected almost universally by linear probing and effective rank.<n>We report a case where both read healthily while the representation carries zero usable instance information.<n>A Graph-JEPA predicts one masked aspect from a subgraph's remaining aspects, attaining linear-probe accuracy 0.871 and effective rank 18-47.<n>We release a harness with a reducibility audit and target gate.
  arXiv  Detail & Related papers  (2026-08-20T19:22:49Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• A Few Neurons Reveal When LLMs Misuse Tools: Sparse Detection and Selective Steering for Reliable Tool Use [28.57781785937644]
  We introduce PRISMS (Probing Representations In Support of Monitoring and Steering), a closed-loop framework that shares a failure-specific neuron basis between sparse detection and activation.<n>Across six models, PRISMS reduces pooled over-calling rate by 80% while increasing tool-required accuracy by 14.2 percentage points.
  arXiv  Detail & Related papers  (2026-07-31T19:03:44Z)
• Are Single-Token Sparse Autoencoder Features Causally Necessary? Layer-Depth and SAE-Family Effects [1.630081999385542]
  Single-token features that activate on one vocabulary item provide the diagnostic case where ground truth permits direct comparison.<n>We analyze 3.9M features across six models and three SAE families using zero-ablation at full layer depth.
  arXiv  Detail & Related papers  (2026-07-22T17:33:16Z)
• From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models [0.0]
  We propose a five-stage methodology for causal feature analysis in transformer language models.<n>We demonstrate it end-to-end on GPT-2 small performing the Indirect Object Identification (IOI) task.
  arXiv  Detail & Related papers  (2026-05-21T13:25:16Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes [53.726365933748134]
  We analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions.<n>We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $sim$70% activation sparsity.
  arXiv  Detail & Related papers  (2026-05-17T21:29:20Z)
• ContractBench: Can LLM Agents Preserve Observation Contracts? [9.057486468322933]
  We show that observation contract compliance is an emergent, regression-prone capability.<n>We introduce ContractBench, a benchmark of 33 dual-axis tasks that probe two failure modes.<n>We report four findings: (i) no evaluated model clears 80%, with Claude-Opus-4.6 leading at 77.8%; (ii) a sharp within-family capability cliff in Qwen 3.5 between 4B (0%) and 9B (56.6%), smoothing to 70.7% at 397B-A17B; (iii) non-monotonic scaling across the GPT-5 family: agentic post-training can erode
  arXiv  Detail & Related papers  (2026-05-17T06:37:04Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• PBT-Bench: Benchmarking AI Agents on Property-Based Testing [29.035258104995204]
  We introduce PBT-Bench, a benchmark of 100 curated property-based testing problems across 40 real Python libraries.<n>Each problem injects one or more semantic bugs (365 in total, mean 3.65 per problem) designed so that default-strategy random inputs almost never trigger them.<n>Bug recall under the PBT-guided prompt ranges from 42.1% to 83.4% across models; under the open-ended baseline, from 31.4% to 76.7%.
  arXiv  Detail & Related papers  (2026-05-13T18:01:05Z)
• Single-Position Intervention Fails: Distributed Output Templates Drive In-Context Learning [0.0]
  Understanding how large language models encode task identity from few-shot demonstrations is a central open problem in mechanistic interpretability.<n>Prior work uses linear probing to localize task representations, reporting high classification accuracy at specific layers.<n>We reveal a striking dissociation: probing accuracy completely fails to predict causal importance.
  arXiv  Detail & Related papers  (2026-04-10T14:49:07Z)
• Steerable but Not Decodable: Function Vectors Operate Beyond the Logit Lens [0.0]
  Function (FV) steering succeeds even when the logit lens cannot decode the correct answer at any layer.<n>FV steering exceeds logit lens accuracy for every task on every model.<n>FV vocabulary projection reveals that FVs achieving over 0.90 steering accuracy still project to incoherent token distributions.
  arXiv  Detail & Related papers  (2026-04-03T00:54:11Z)
• Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents [0.25891009512378166]
  Chain-of-thought (CoT) reasoning is widely assumed to improve agent performance.<n>We present a systematic study of CoT budget effects on function-calling agents.<n>Our central finding is a striking non-monotonic pattern on Qwen2.5-1.5B-Instruct: brief reasoning (32 tokens) dramatically improves accuracy by 45% relative over direct answers.
  arXiv  Detail & Related papers  (2026-04-02T15:25:13Z)
• Benchmarking Reasoning Robustness in Large Language Models [76.79744000300363]
  We find significant performance degradation on novel or incomplete data.<n>These findings highlight the reliance on recall over rigorous logical inference.<n>This paper introduces a novel benchmark, termed as Math-RoB, that exploits hallucinations triggered by missing information to expose reasoning gaps.
  arXiv  Detail & Related papers  (2025-03-06T15:36:06Z)
• SCOP: Scientific Control for Reliable Neural Network Pruning [127.20073865874636]
  This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Redundant filters can be discovered in the adversarial process of different features. Our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet.
  arXiv  Detail & Related papers  (2020-10-21T03:02:01Z)
• HRank: Filter Pruning using High-Rank Feature Map [149.86903824840752]
  We propose a novel filter pruning method by exploring the High Rank of feature maps (HRank) Our HRank is inspired by the discovery that the average rank of multiple feature maps generated by a single filter is always the same. Based on HRank, we develop a method that is mathematically formulated to prune filters with low-rank feature maps.
  arXiv  Detail & Related papers  (2020-02-24T11:50:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.