PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks

• URL: http://arxiv.org/abs/2603.13347v1
• Date: Sat, 07 Mar 2026 10:39:56 GMT
• Title: PolyGLU: State-Conditional Activation Routing in Transformer Feed-Forward Networks
• Abstract summary: We introduce PolyGLU, a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions.<n>On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Biological neural systems employ diverse neurotransmitters -- glutamate, GABA, dopamine, acetylcholine -- to implement distinct signal-processing modalities within shared neural circuits. In contrast, modern transformers apply a single fixed activation function across all feed-forward neurons. We introduce PolyGLU (Polychromatic Gated Linear Unit), a drop-in replacement for SwiGLU that enables each FFN neuron to dynamically route among K=4 activation functions via a differentiable mechanism combining learned static preferences with input-conditioned gating, trained end-to-end with Gumbel-Softmax. We train PolychromaticLM, a 597M-parameter transformer, on ~10B tokens using a single NVIDIA A100 GPU. Our key finding is emergent routing behavior: without any explicit sparsity loss or entropy regularization, the routing mechanism converges to near-deterministic activation selections (mean dynamic entropy = 0.030% of maximum), with a striking depth-dependent specialization pattern -- early layers prefer GELU while deep layers strongly favor Tanh. Three layers maintain elevated routing entropy, suggesting computational flexibility points. The routing architecture adds only 0.23% parameter overhead (~1.4M parameters) and proves fully robust to supervised fine-tuning: routing entropy remains constant at ln(4) throughout 13,067 SFT steps. On standard benchmarks, PolychromaticLM achieves 62-89% of Qwen3-0.6B-Base performance despite training on 3,600x fewer tokens. All code, weights, and training infrastructure are released under Apache 2.0.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Machine Learning-Based Reconstruction for Resistive Silicon Sensors [53.365453214937496]
  We study machine-learning-based reconstruction and compression for resistive silicon sensors.<n>We use full-waveform information from correlated pads to regularise the reconstruction and extract spatial information.<n>We first introduce recurrent neural network models based on LSTM layers, which provide a proof-of-concept implementation for full-waveform reconstruction.
  arXiv  Detail & Related papers  (2026-07-13T14:03:26Z)
• Structural Kolmogorov-Arnold Convolutions: Learnable Function on the Values or the Filter Shape as Parameter-Efficient Alternative to Per-Edge Convolutional KANs [76.06235645266621]
  Convolutional Kolmogorov--Arnold Networks (KANs) replace the fixed weights of a convolutional kernel with learnable univariate functions.<n>We argue that the learnable functions are better placed in the emphstructure of the convolution than on each edge.
  arXiv  Detail & Related papers  (2026-06-23T10:00:26Z)
• SNAPO: Smooth Neural Adjoint Policy Optimization for Optimal Control via Differentiable Simulation [0.0]
  SNAPO is a framework that embeds a neural policy inside a known, differentiable simulator.<n>It computes exact gradients of the objective with respect to all policy parameters and all inputs in a single adjoint pass.<n>We demonstrate SNAPO on three domains: natural gas storage (training in under a minute, 365 forward curve sensitivities at no additional cost per sensitivity), pension fund asset-liability management, and pharmaceutical manufacturing.
  arXiv  Detail & Related papers  (2026-05-07T17:01:13Z)
• Training Non-Differentiable Networks via Optimal Transport [1.5229257192293202]
  PolyStep is a gradient-free method that updates parameters using only forward passes.<n>We train genuinely non-differentiable models where existing gradient-free methods collapse to near-random accuracy.<n>On hard-LIF spiking networks we reach 93.4% test accuracy, outperforming all gradient-free baselines by over 60pp and closing to within 4.4pp of a surrogate-gradient Adam ceiling.
  arXiv  Detail & Related papers  (2026-05-03T15:20:56Z)
• Metriplector: From Field Theory to Neural Architecture [0.25168553347063854]
  We present Metriplector, a neural architecture primitive in which the input configures an abstract physical system.<n>Multiple fields evolve via coupled metriplectic dynamics, and the stress-energy tensor T, derived from Noether's theorem, provides the readout.<n>We evaluate Metriplector across five domains, each using a task-specific architecture built from this shared primitive with progressively richer physics.
  arXiv  Detail & Related papers  (2026-03-31T09:40:26Z)
• The Discrete Charm of the MLP: Binary Routing of Continuous Signals in Transformer Feed-Forward Layers [0.0]
  We show that layers in language models perform binary routing of continuous signals.<n>We find that specific neurons implement a consensus architecture that are 93-98% mutually exclusive.<n>We propose that the well-established piecewise-affine characterization of deep networks can be complemented by a routing characterization.
  arXiv  Detail & Related papers  (2026-03-11T17:14:57Z)
• Differentiable Logic Synthesis: Spectral Coefficient Selection via Sinkhorn-Constrained Composition [0.0]
  We introduce Hierarchical Spectral Composition, a differentiable architecture that selects spectral coefficients from a frozen Fourier basis.<n>We adapt this framework to logic synthesis, adding column-sign modulation to enable Boolean negation.
  arXiv  Detail & Related papers  (2026-01-20T13:26:52Z)
• Spiking Decision Transformers: Local Plasticity, Phase-Coding, and Dendritic Routing for Low-Power Sequence Control [0.6117371161379209]
  Spiking neural networks promise ultra-low-power, event-driven inference.<n>We present the Spiking Decision Transformer (SNN-DT), which embeds Leaky Integrate-and-Fire neurons into each self-attention block.<n>By marrying sequence modeling with neuromorphic efficiency, SNN-DT opens a pathway toward real-time, low-power control on embedded and wearable devices.
  arXiv  Detail & Related papers  (2025-08-29T10:37:37Z)
• Tensor Decomposition Networks for Fast Machine Learning Interatomic Potential Computations [48.46721044282335]
  tensor decomposition networks (TDNs) achieve competitive performance with dramatic speedup in computations.<n>We evaluate TDNs on PubChemQCR, a newly curated molecular relaxation dataset containing 105 million DFT-calculated snapshots.<n>Results show that TDNs achieve competitive performance with dramatic speedup in computations.
  arXiv  Detail & Related papers  (2025-07-01T18:46:27Z)
• LipKernel: Lipschitz-Bounded Convolutional Neural Networks via Dissipative Layers [0.0468732641979009]
  We propose a layer-wise parameterization for convolutional neural networks (CNNs) that includes built-in robustness guarantees. Our method Lip Kernel directly parameterizes dissipative convolution kernels using a 2-D Roesser-type state space model. We show that the run-time using our method is orders of magnitude faster than state-of-the-art Lipschitz-bounded networks.
  arXiv  Detail & Related papers  (2024-10-29T17:20:14Z)
• Kolmogorov-Arnold Transformer [72.88137795439407]
  We introduce the Kolmogorov-Arnold Transformer (KAT), a novel architecture that replaces layers with Kolmogorov-Arnold Network (KAN) layers. We identify three key challenges: (C1) Base function, (C2) Inefficiency, and (C3) Weight. With these designs, KAT outperforms traditional-based transformers.
  arXiv  Detail & Related papers  (2024-09-16T17:54:51Z)
• Uniformer: Unified Transformer for Efficient Spatiotemporal Representation Learning [68.55487598401788]
  Recent advances in this research have been mainly driven by 3D convolutional neural networks and vision transformers. We propose a novel Unified transFormer (UniFormer) which seamlessly integrates merits of 3D convolution self-attention in a concise transformer format. We conduct extensive experiments on the popular video benchmarks, e.g., Kinetics-400, Kinetics-600, and Something-Something V1&V2. Our UniFormer achieves 8/84.8% top-1 accuracy on Kinetics-400/Kinetics-600, while requiring 10x fewer GFLOPs than other state-of-the-art methods
  arXiv  Detail & Related papers  (2022-01-12T20:02:32Z)
• DS-Net++: Dynamic Weight Slicing for Efficient Inference in CNNs and Transformers [105.74546828182834]
  We show a hardware-efficient dynamic inference regime, named dynamic weight slicing, which adaptively slice a part of network parameters for inputs with diverse difficulty levels. We present dynamic slimmable network (DS-Net) and dynamic slice-able network (DS-Net++) by input-dependently adjusting filter numbers of CNNs and multiple dimensions in both CNNs and transformers.
  arXiv  Detail & Related papers  (2021-09-21T09:57:21Z)
• Dynamic Convolution for 3D Point Cloud Instance Segmentation [146.7971476424351]
  We propose an approach to instance segmentation from 3D point clouds based on dynamic convolution. We gather homogeneous points that have identical semantic categories and close votes for the geometric centroids. The proposed approach is proposal-free, and instead exploits a convolution process that adapts to the spatial and semantic characteristics of each instance.
  arXiv  Detail & Related papers  (2021-07-18T09:05:16Z)
• Dynamic Multi-Branch Layers for On-Device Neural Machine Translation [53.637479651600586]
  We propose to improve the performance of on-device neural machine translation (NMT) systems with dynamic multi-branch layers. Specifically, we design a layer-wise dynamic multi-branch network with only one branch activated during training and inference. At almost the same computational cost, our method achieves improvements of up to 1.7 BLEU points on the WMT14 English-German translation task and 1.8 BLEU points on the WMT20 Chinese-English translation task.
  arXiv  Detail & Related papers  (2021-05-14T07:32:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.