Training, Reading, and Editing Legible Transformers

• URL: http://arxiv.org/abs/2607.08946v1
• Date: Thu, 09 Jul 2026 21:15:40 GMT
• Title: Training, Reading, and Editing Legible Transformers
• Abstract summary: A transformer can be built from operators that are legible by construction.<n>A crispness penalty meant to sharpen a bounded operator into a decisive detector instead collapses it into a dead constant.<n>A learned per-unit fraction then retires the hand-set reserved-GELU partition of prior work.
• Score: 0.2663471820643486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A transformer can be built from operators that are legible by construction -- bounded, named units that read as fuzzy set operations rather than dense activations -- but legibility must be pressed for during training, and the pressure has a failure mode. A crispness penalty meant to sharpen a bounded operator into a decisive detector instead collapses it into a dead constant. An identity, E[v(1-v)] = mu(1-mu) - var, shows why -- the penalty is a variance-minimizer blind to the difference between a live detector and a constant -- and names the fix: a per-channel variance floor, the target legibility metric written as a loss, which recovers both legibility and quality. A learned per-unit fraction then retires the hand-set reserved-GELU partition of prior work: given the choice the model keeps no unit as pure GELU and routes 87% of its load-bearing computation through crisp operators. The result is the most legible transformer we have built -- 78% of its feed-forward operands and 50% of its attention value channels are crisp-and-contextual detectors, and per-head legibility rises from 18% in shallow layers to 78% in deep ones. Read in the correct rotated per-layer frame, these units separate a clean detection (what a unit responds to) from a harder naming (what its output decodes to); and because the objective makes each unit crisp and sparse, edits to them are far more local -- 50-184x in the deep layers where the edit sites concentrate -- and can target explicit conjunctions a single neuron cannot express. Finally, a between-unit decorrelation pressure exposes a legibility dial: it trades a circuit's reuse for independence at no quality cost, turning concepts into single, surgically editable units and a prediction into a short explanation read off a handful of named operations. Quality holds at parity with a conventional baseline throughout.

Related papers

• Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions [87.95757610455173]
  We introduce an injectively aligned leave-one-cell-out protocol over support x operation grids and SO-OPF.<n>We separate two questions that aggregate scores conflate: whether the carrier composes held-out bindings when the grid is known, and whether that grid can be recovered from flat cell labels.
  arXiv  Detail & Related papers  (2026-08-06T15:38:55Z)
• Legible-by-Construction: Attention and End-to-End Transformers [0.2663471820643486]
  A transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations.<n> hidden units read as named logical operators at no cost to language-model quality.<n>Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become legible.
  arXiv  Detail & Related papers  (2026-07-05T14:08:59Z)
• Conditional Co-Ablation: Recovering Self-Repair Backups in Transformer Circuits [47.668752416295185]
  Conditional Co-Ablation asks how much each remaining unit's ablation effect grows once a primary set has been removed.<n>On the GPT-2-small IOI circuit, CoAx raises backup-head recovery from 0.33 to 0.91 ROC-AUC.
  arXiv  Detail & Related papers  (2026-07-02T09:32:57Z)
• Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors [0.2663471820643486]
  NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form.<n>We resolve both with a small block of sequence quantifiers.<n>The result is a parameter-neutral, language-model-quality transformer.
  arXiv  Detail & Related papers  (2026-06-30T15:46:27Z)
• Output Vector Editing for Memorization Mitigation in Large Language Models [68.30351930772788]
  Large language models memorize and reproduce sequences from their training data, creating privacy, copyright, and security risks.<n>Existing neuron-level mitigation methods equate editing with zeroing out neuron activations, but the activation only controls whether a neuron engages.<n>We propose output vector editing, a constrained-optimization edit that locates a small set of neurons responsible for a memorized continuation.
  arXiv  Detail & Related papers  (2026-06-17T07:29:18Z)
• Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses [57.20181537213498]
  Agentic Harness Engineering (AHE) is a closed loop that automates harness engineering.<n>AHE addresses challenges through three matched observability pillars.<n>Ten AHE lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI.<n>On SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains.
  arXiv  Detail & Related papers  (2026-04-28T16:55:02Z)
• Transformers Can Learn Rules They've Never Seen: Proof of Computation Beyond Interpolation [0.0]
  We test a strong-only hypothesis in two controlled settings.<n>In Experiment 1, we use a cellular automaton with a pure XOR transition rule.<n>In Experiment 2, we study symbolic operator chains over integers with one operator pair held out.
  arXiv  Detail & Related papers  (2026-03-17T18:02:28Z)
• To Throw a Stone with Six Birds: On Agents and Agenthood [0.0]
  Six Birds Theory (SBT) treats macroscopic objects as induced closures rather than primitives.<n>We give a type-correct account of agency within SBT.<n>We operationalize this contract in finite controlled systems using four checkable components.
  arXiv  Detail & Related papers  (2026-02-03T10:46:23Z)
• GeneralAD: Anomaly Detection Across Domains by Attending to Distorted Features [68.14842693208465]
  GeneralAD is an anomaly detection framework designed to operate in semantic, near-distribution, and industrial settings. We propose a novel self-supervised anomaly generation module that employs straightforward operations like noise addition and shuffling to patch features. We extensively evaluated our approach on ten datasets, achieving state-of-the-art results in six and on-par performance in the remaining.
  arXiv  Detail & Related papers  (2024-07-17T09:27:41Z)
• Mapping of attention mechanisms to a generalized Potts model [50.91742043564049]
  We show that training a neural network is exactly equivalent to solving the inverse Potts problem by the so-called pseudo-likelihood method. We also compute the generalization error of self-attention in a model scenario analytically using the replica method.
  arXiv  Detail & Related papers  (2023-04-14T16:32:56Z)
• HEAT: Hardware-Efficient Automatic Tensor Decomposition for Transformer Compression [69.36555801766762]
  We propose a hardware-aware tensor decomposition framework, dubbed HEAT, that enables efficient exploration of the exponential space of possible decompositions. We experimentally show that our hardware-aware factorized BERT variants reduce the energy-delay product by 5.7x with less than 1.1% accuracy loss.
  arXiv  Detail & Related papers  (2022-11-30T05:31:45Z)
• Delta Keyword Transformer: Bringing Transformers to the Edge through Dynamically Pruned Multi-Head Self-Attention [4.925939498983408]
  Multi-head self-attention forms the core of Transformer networks. We propose a dynamic pruning method, which exploits the temporal stability of data across tokens to reduce inference cost. Our experiments show that we can reduce 80% of operations while maintaining the original 98.4% accuracy.
  arXiv  Detail & Related papers  (2022-03-20T20:59:13Z)
• Fault-tolerant parity readout on a shuttling-based trapped-ion quantum computer [64.47265213752996]
  We experimentally demonstrate a fault-tolerant weight-4 parity check measurement scheme. We achieve a flag-conditioned parity measurement single-shot fidelity of 93.2(2)%. The scheme is an essential building block in a broad class of stabilizer quantum error correction protocols.
  arXiv  Detail & Related papers  (2021-07-13T20:08:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.