Legible-by-Construction: Attention and End-to-End Transformers

• URL: http://arxiv.org/abs/2607.04319v1
• Date: Sun, 05 Jul 2026 14:08:59 GMT
• Title: Legible-by-Construction: Attention and End-to-End Transformers
• Abstract summary: A transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations.<n> hidden units read as named logical operators at no cost to language-model quality.<n>Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become legible.
• Score: 0.2663471820643486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A companion paper showed that a transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations - intersection, set-difference, and a self-forgetting sequence quantifier - so its hidden units read as named logical operators at no cost to language-model quality. That left the other half of the transformer opaque. Here we carry the same idea into attention and join the two into one model. The mechanism is minimal: a head's value is passed through a sigmoid, so each value channel becomes a readable detector of whether a feature holds at a token. This adds no parameters and leaves the standard head otherwise untouched. A Boolean variant goes further, restructuring the value into an explicit within-token intersection and negation-capable set-difference. In both designs the output projection is left free, not tied to the vocabulary, which is the load-bearing decision: bounding what a head detects while leaving what it writes unconstrained yields selective detectors, whereas constraining the write does not. A bounded value is shaped into a readable detector by two selectivity pressures - one for sparse firing, one for decisive firing at the rails - and which a design wants is not universal. Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become crisp, contextually selective detectors, and their legibility rises with depth rather than crystallizing only on punctuation. Language-model quality is at parity with a conventional baseline. Finally, we couple the Boolean attention to the legible feed-forward layer and train an end-to-end legible-by-construction language model at benchmark parity: its feed-forward units are named set and quantifier operations throughout, and we can take a token it generates and read the named units that compose to produce it.

Related papers

• StateBridge: Training-free Hidden-state Alignment for Latent Communication in LLM Multi-Agent Systems [35.74603358195608]
  We propose a training-free latent communication approach that aligns the sender's hidden states to the receiver's input space via a closed-form transformation.<n>StateBridge achieves the best or tied-best score on 22 out of 26 model-task pairs.
  arXiv  Detail & Related papers  (2026-08-13T14:40:59Z)
• Training, Reading, and Editing Legible Transformers [0.2663471820643486]
  A transformer can be built from operators that are legible by construction.<n>A crispness penalty meant to sharpen a bounded operator into a decisive detector instead collapses it into a dead constant.<n>A learned per-unit fraction then retires the hand-set reserved-GELU partition of prior work.
  arXiv  Detail & Related papers  (2026-07-09T21:15:40Z)
• Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors [0.2663471820643486]
  NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form.<n>We resolve both with a small block of sequence quantifiers.<n>The result is a parameter-neutral, language-model-quality transformer.
  arXiv  Detail & Related papers  (2026-06-30T15:46:27Z)
• The Deterministic Horizon: Impossibility Results as Design Specifications for Trustworthy AI Systems [0.0]
  This thesis turns impossibility results from curiosities into design rules.<n>Its flagship result proves an accuracy ceiling set by architecture alone.<n>The same argument recasts across subfields.
  arXiv  Detail & Related papers  (2026-05-21T20:48:35Z)
• Direct Semantic Communication Between Large Language Models via Vector Translation [3.81908263930559]
  Large language models (LLMs) pass messages as plain tokens, discarding most latent semantics.<n>We form a latent bridge via vector translations, which use learned mappings that enable direct semantic exchange.
  arXiv  Detail & Related papers  (2025-11-06T00:43:29Z)
• Semantic Fusion with Fuzzy-Membership Features for Controllable Language Modelling [0.0]
  semantic fusion is a lightweight scheme that augments a Transformer language model (LM) with a fuzzy-membership feature channel.<n>Each token is represented by a vector of interpretable features whose values are graded degrees from differentiable membership functions.<n>This approach adds only small overhead, remains fully compatible with tied input-output embeddings, and provides an interpretable pathway for conditioned natural language generation.
  arXiv  Detail & Related papers  (2025-09-14T22:11:09Z)
• Evaluating Semantic Variation in Text-to-Image Synthesis: A Causal Perspective [50.261681681643076]
  We propose a novel metric called SemVarEffect and a benchmark named SemVarBench to evaluate the causality between semantic variations in inputs and outputs in text-to-image synthesis.<n>Our work establishes an effective evaluation framework that advances the T2I synthesis community's exploration of human instruction understanding.
  arXiv  Detail & Related papers  (2024-10-14T08:45:35Z)
• Activation Scaling for Steering and Interpreting Language Models [55.59689963561315]
  We argue that successfully intervening on a model is a prerequisite for interpreting its internal workings. We establish a three-term objective: a successful intervention should flip the correct with the wrong token and vice versa. Using gradient-based optimization, this objective lets us learn (and later evaluate) a specific kind of efficient and interpretable intervention.
  arXiv  Detail & Related papers  (2024-10-07T12:01:32Z)
• Autoregressive Speech Synthesis without Vector Quantization [135.4776759536272]
  We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS)<n>MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition.<n>MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes.
  arXiv  Detail & Related papers  (2024-07-11T14:36:53Z)
• Shapley Head Pruning: Identifying and Removing Interference in Multilingual Transformers [54.4919139401528]
  We show that it is possible to reduce interference by identifying and pruning language-specific parameters. We show that removing identified attention heads from a fixed model improves performance for a target language on both sentence classification and structural prediction.
  arXiv  Detail & Related papers  (2022-10-11T18:11:37Z)
• Fast End-to-End Speech Recognition via a Non-Autoregressive Model and Cross-Modal Knowledge Transferring from BERT [72.93855288283059]
  We propose a non-autoregressive speech recognition model called LASO (Listen Attentively, and Spell Once) The model consists of an encoder, a decoder, and a position dependent summarizer (PDS)
  arXiv  Detail & Related papers  (2021-02-15T15:18:59Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Stacked DeBERT: All Attention in Incomplete Data for Text Classification [8.900866276512364]
  We propose Stacked DeBERT, short for Stacked Denoising Bidirectional Representations from Transformers. Our model shows improved F1-scores and better robustness in informal/incorrect texts present in tweets and in texts with Speech-to-Text error in sentiment and intent classification tasks.
  arXiv  Detail & Related papers  (2020-01-01T04:49:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.