The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers

• URL: http://arxiv.org/abs/2603.10985v1
• Date: Wed, 11 Mar 2026 17:14:57 GMT
• Title: The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers
• Abstract summary: We show that layers in language models perform binary routing of continuous signals.<n>We find that specific neurons implement a consensus architecture that are 93-98% mutually exclusive.<n>We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that MLP layers in transformer language models perform binary routing of continuous signals: the decision of whether a token needs nonlinear processing is well-captured by binary neuron activations, even though the signals being routed are continuous. In GPT-2 Small (124M parameters), we find that specific neurons implement a consensus architecture -- seven "default-ON" neurons and one exception handler (N2123 in Layer 11) that are 93-98% mutually exclusive -- creating a binary routing switch. A cross-layer analysis reveals a developmental arc: early layers (L1-3) use single gateway neurons to route exceptions without consensus quorums; middle layers (L4-6) show diffuse processing with neither gateway nor consensus; and late layers (L7-11) crystallize full consensus/exception architectures with increasing quorum size (1 to 3 to 7 consensus neurons). Causal validation confirms the routing is functional: removing the MLP at consensus breakdown costs 43.3% perplexity, while at full consensus removing it costs only 10.1% -- exceeding a 4x difference. Comparing binary vs. continuous features for the routing decision confirms that binarization loses essentially no information (79.2% vs. 78.8% accuracy), while continuous activations carry additional magnitude information (R^2 = 0.36 vs. 0.22). This binary routing structure explains why smooth polynomial approximation fails: cross-validated polynomial fits (degrees 2-7) never exceed R^2 = 0.06 for highly nonlinear layers. We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization: along the natural data manifold, the piecewise boundaries implement binary decisions about which tokens need nonlinear processing, routing continuous signals through qualitatively different computational paths.

Related papers

• A*-Thought-V2: Efficient Latent Reasoning via Geometric Dynamics of LLM [40.07171947823021]
  Chain-of-Thought (CoT) improves the reasoning ability of Large Language Models (LLMs) but incurs substantial computation and context costs.<n>We present A*-Thought-V2, a geometric dynamics of LLM guided framework that models CoT as a hidden-state trajectory and replaces hard deletion with an explicit-implicit interleaved latent architecture.
  arXiv  Detail & Related papers  (2026-09-07T17:56:20Z)
• Long-Range Indirect Control-Flow Prediction in Stripped Binaries via Dual Virtual Hubs and Multi-Task Graph Learning [44.79384874347874]
  We present ICFlowNet, a unified framework for long-range ICF prediction in stripped binaries.<n>ICFlowNet introduces candidate-aware Dual Virtual Hubs, a Global Code Hub and a Global Data Hub.<n>We also develop a leakage-aware, noise-controlled pipeline with package-level splits, function-level mnemonic-hash deduplication, and a clean test protocol.
  arXiv  Detail & Related papers  (2026-09-03T02:17:23Z)
• NeuronSoup: Evolving Asynchronous, Shared-Neuron Temporal Graphs without Backpropagation [0.8122270502556375]
  We present NeuronSoup, a neural computation architecture that replaces synchronous layer-by-layer processing with asynchronous, delay-mediated signal propagation.<n>The entire architecture is co-evolved by a genetic algorithm operating on a flat real-valued genome of 14,602 genes.<n>On 10-class MNIST digit classification using frozen ResNet18 features as input, the system evolves a network of 204 active paths through 266 hidden neurons.
  arXiv  Detail & Related papers  (2026-07-16T17:18:59Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Perturbation Probing: A Two-Pass-per-Prompt Diagnostic for FFN Behavioral Circuits in Aligned LLMs [9.127363793428119]
  Perturbation probing generates task-specific causal hypotheses for FFN neurons in large language models.<n>Across eight behavioral circuits, 13 models, and four architecture families, we identify two circuit structures that organize LLM behavior.
  arXiv  Detail & Related papers  (2026-04-30T04:13:33Z)
• Darkness Visible: Reading the Exception Handler of a Language Model [0.0]
  GPT-2 uses verb subcategorization immediately, consistent with the exception handler operating at token-level predictability rather than syntactic structure.<n>This architecture crystallizes only at the terminal layer -- in deeper models, we predict equivalent structure at the final layer, not at layer 11.
  arXiv  Detail & Related papers  (2026-04-06T15:24:05Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks [0.0]
  We introduce PolyGLU, a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions.<n>On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens.
  arXiv  Detail & Related papers  (2026-03-07T10:39:56Z)
• Polynomial Surrogate Training for Differentiable Ternary Logic Gate Networks [3.2249474972573555]
  We introduce Polynomial Surrogate Training (PST), which represents each ternary neuron as a degree-$ (2,2)$ with 9 learnable coefficients.<n>PST trains $2$-$3times$ faster than binary DLGNs and discover true ternary gates that are functionally diverse.
  arXiv  Detail & Related papers  (2026-02-27T20:37:17Z)
• Generalizing GNNs with Tokenized Mixture of Experts [75.8310720413187]
  We show that improving stability requires reducing reliance on shift-sensitive features, leaving an irreducible worst-case generalization floor.<n>We propose STEM-GNN, a pretrain-then-finetune framework with a mixture-of-experts encoder for diverse computation paths.<n>Across nine node, link, and graph benchmarks, STEM-GNN achieves a stronger three-way balance, improving robustness to degree/homophily shifts and to feature/edge corruptions while remaining competitive on clean graphs.
  arXiv  Detail & Related papers  (2026-02-09T22:48:30Z)
• Differentiable Logic Synthesis: Spectral Coefficient Selection via Sinkhorn-Constrained Composition [0.0]
  We introduce Hierarchical Spectral Composition, a differentiable architecture that selects spectral coefficients from a frozen Fourier basis.<n>We adapt this framework to logic synthesis, adding column-sign modulation to enable Boolean negation.
  arXiv  Detail & Related papers  (2026-01-20T13:26:52Z)
• Automated Circuit Interpretation via Probe Prompting [0.0]
  We present probe prompting, an automated pipeline that transforms attribution graphs into compact, interpretable subgraphs.<n>Across five prompts, probe-prompted subgraphs preserve high explanatory coverage while compressing complexity.<n>Compared to geometric clustering baselines, concept-aligned groups exhibit higher behavioral coherence.
  arXiv  Detail & Related papers  (2025-11-10T11:53:36Z)
• Environment-Aware Indoor LoRaWAN Path Loss: Parametric Regression Comparisons, Shadow Fading, and Calibrated Fade Margins [3.776919981139063]
  Indoor LoRaWAN propagation is shaped by structural and time-varying context factors.<n>We present an environment-aware, statistically disciplined path loss framework evaluated using leakage-safe cross-validation.
  arXiv  Detail & Related papers  (2025-10-05T20:14:48Z)
• Robust Residual Finite Scalar Quantization for Neural Compression [46.574899938569125]
  Finite Scalar Quantization (FSQ) offers simplified training but suffers from residual magnitude decay in multi-stage settings.<n>We propose Robust Residual Finite Scalar Quantization (RFSQ), addressing this fundamental limitation through two novel conditioning strategies.<n>Our experiments across audio and image modalities demonstrate RFSQ's effectiveness and generalizability.
  arXiv  Detail & Related papers  (2025-08-20T15:18:59Z)
• Towards Accurate Binary Neural Networks via Modeling Contextual Dependencies [52.691032025163175]
  Existing Binary Neural Networks (BNNs) operate mainly on local convolutions with binarization function. We present new designs of binary neural modules, which enables leading binary neural modules by a large margin.
  arXiv  Detail & Related papers  (2022-09-03T11:51:04Z)
• Bayesian Neural Network Language Modeling for Speech Recognition [59.681758762712754]
  State-of-the-art neural network language models (NNLMs) represented by long short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming highly complex. In this paper, an overarching full Bayesian learning framework is proposed to account for the underlying uncertainty in LSTM-RNN and Transformer LMs.
  arXiv  Detail & Related papers  (2022-08-28T17:50:19Z)
• Non-Gradient Manifold Neural Network [79.44066256794187]
  Deep neural network (DNN) generally takes thousands of iterations to optimize via gradient descent. We propose a novel manifold neural network based on non-gradient optimization.
  arXiv  Detail & Related papers  (2021-06-15T06:39:13Z)
• Learning Frequency Domain Approximation for Binary Neural Networks [68.79904499480025]
  We propose to estimate the gradient of sign function in the Fourier frequency domain using the combination of sine functions for training BNNs. The experiments on several benchmark datasets and neural architectures illustrate that the binary network learned using our method achieves the state-of-the-art accuracy.
  arXiv  Detail & Related papers  (2021-03-01T08:25:26Z)
• When Residual Learning Meets Dense Aggregation: Rethinking the Aggregation of Deep Neural Networks [57.0502745301132]
  We propose Micro-Dense Nets, a novel architecture with global residual learning and local micro-dense aggregations. Our micro-dense block can be integrated with neural architecture search based models to boost their performance.
  arXiv  Detail & Related papers  (2020-04-19T08:34:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.