Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors

• URL: http://arxiv.org/abs/2606.31845v1
• Date: Tue, 30 Jun 2026 15:46:27 GMT
• Title: Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors
• Abstract summary: NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form.<n>We resolve both with a small block of sequence quantifiers.<n>The result is a parameter-neutral, language-model-quality transformer.
• Score: 0.2663471820643486
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A transformer's feed-forward (FFN) sublayer materializes the distinctions attention gathers, yet gives no account of what it computes. In a parameter-neutral replacement, each hidden unit is an explicit fuzzy set operation on sigmoid-bounded [0,1] memberships: intersection A*B and set-difference A*(1-B), the latter a bounded positive negation ("A but not B") that gated/bilinear units lack -- a negation-capable FFN (NC-FFN). On N-bit parity they are the most parameter-efficient reasoning basis at shallow depth; at scale (125M, OpenWebText) NC-FFN ties the GELU baseline's perplexity, every unit carrying explicit logical form. Two limits share one cause: two-operand logic localizes to layer 0 and erodes under training, and the one robust grammatical deficit concentrates in licensing and quantifiers, beyond within-token operators. We resolve both with a small block of sequence quantifiers: a soft existential and a soft proportion, each with a per-unit learned forgetting rate from a sticky init. This recovers the deficit at epoch one (halving the wider epoch-two gap), modestly leads on LAMBADA, and makes the FFN legible: the structure now holds and migrates into depth; the decay un-learns its stickiness (median half-life ~1.5 tokens; zero latch units); and at the semantic layers the units read, without dictionary learning, as grammatical licensing detectors: each fires on a licensor (a comparative, a passive participle, a negative-polarity item) and carries its memory forward to predict the licensed word (than, by, nor). This legibility is localized and free only up to a partition (a fully Boolean FFN diverges in training), but the result is a parameter-neutral, language-model-quality transformer with a readable, interpretable-by-construction grammatical mechanism -- an account not just of what a feed-forward layer represents but how it licenses.

Related papers

• What Does a Discrete Diffusion Model Learn? [71.03603607324338]
  discrete diffusion model learns: a denoiser, a score ratio, or a bridge plug-in predictor.<n>We start with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process.<n>Every identity is verified numerically, without approximation, on an exactly solvable model.
  arXiv  Detail & Related papers  (2026-07-06T17:56:11Z)
• Legible-by-Construction: Attention and End-to-End Transformers [0.2663471820643486]
  A transformer's feed-forward layer can be rebuilt from explicit fuzzy set operations.<n> hidden units read as named logical operators at no cost to language-model quality.<n>Across five specialized-attention designs at 125M parameters, 44 to 62 percent of value channels become legible.
  arXiv  Detail & Related papers  (2026-07-05T14:08:59Z)
• Weave of Formal Thought [51.56484100374058]
  We introduce Weave of Formal Thought (WoFT), a paradigm uniting rigorous syntactic validation with learned structural representations.<n>We present a latent-variable fine-tuning method training the language model to interleave non-terminal grammar symbols directly into generation.<n>For Python, fine-tuning StarCoder2-3B with our RWS objective reduces per-token cross-entropy by 14.3% relative to a text-only SFT baseline.
  arXiv  Detail & Related papers  (2026-06-24T15:58:11Z)
• TIDE: Every Layer Knows the Token Beneath the Context [21.609102126208608]
  We propose TIDE, which augments the standard transformer with EmbeddingMemory.<n>We theoretically and empirically establish the benefits of TIDE in addressing the issues associated with single-token identity injection.
  arXiv  Detail & Related papers  (2026-05-07T13:16:18Z)
• Learning with Boolean threshold functions [0.013714053458441644]
  We develop a method for training neural networks on sparse data in which the values at all nodes are strictly $pm 1$.<n>Across a range of tasks, this method provides a viable and efficient foundation for learning in neural systems.
  arXiv  Detail & Related papers  (2026-02-19T16:07:25Z)
• Grammar-Forced Translation of Natural Language to Temporal Logic using LLMs [11.37102335479887]
  We propose a framework for NL to TL translation called Grammar Forced Translation (GraFT)<n>GraFT reduces the complexity of both tasks by restricting the set of valid output tokens from the full vocabulary to only a handful in each step.<n>We evaluate the effectiveness of GraFT using the CW, GLTL, and Navi benchmarks.
  arXiv  Detail & Related papers  (2025-12-18T17:55:15Z)
• Universal Adversarial Suffixes Using Calibrated Gumbel-Softmax Relaxation [9.099589602551573]
  We study universal adversarial suffixes that, when appended to any input, broadly reduce accuracy across tasks and models.<n>Our approach learns the suffix in a differentiable "soft" form using Gumbel-Softmax relaxation and then discretizes it for inference.<n>A single suffix trained on one model transfers effectively to others, consistently lowering both accuracy and calibrated confidence.
  arXiv  Detail & Related papers  (2025-12-09T00:03:39Z)
• Semantic Fusion with Fuzzy-Membership Features for Controllable Language Modelling [0.0]
  semantic fusion is a lightweight scheme that augments a Transformer language model (LM) with a fuzzy-membership feature channel.<n>Each token is represented by a vector of interpretable features whose values are graded degrees from differentiable membership functions.<n>This approach adds only small overhead, remains fully compatible with tied input-output embeddings, and provides an interpretable pathway for conditioned natural language generation.
  arXiv  Detail & Related papers  (2025-09-14T22:11:09Z)
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding [60.06816407728172]
  Discrete diffusion language models have shown strong potential for text generation.<n>Standard supervised fine-tuning misaligns with semi-autoregressive inference.<n>We propose Blockwise SFT, which partitions responses into fixed-size blocks.
  arXiv  Detail & Related papers  (2025-08-27T02:49:33Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Not all tokens are created equal: Perplexity Attention Weighted Networks for AI generated text detection [49.15148871877941]
  Next-token distribution outputs offer a theoretically appealing approach for detection of large language models (LLMs)<n>We propose the Perplexity Attention Weighted Network (PAWN), which uses the last hidden states of the LLM and positions to weight the sum of a series of features based on metrics from the next-token distribution across the sequence length.<n>PAWN shows competitive and even better performance in-distribution than the strongest baselines with a fraction of their trainable parameters.
  arXiv  Detail & Related papers  (2025-01-07T17:00:49Z)
• Extracting Grammars from a Neural Network Parser for Anomaly Detection in Unknown Formats [79.6676793507792]
  Reinforcement learning has recently shown promise as a technique for training an artificial neural network to parse sentences in some unknown format. This paper presents procedures for extracting production rules from the neural network, and for using these rules to determine whether a given sentence is nominal or anomalous.
  arXiv  Detail & Related papers  (2021-07-30T23:10:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.