Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality

• URL: http://arxiv.org/abs/2604.14419v1
• Date: Wed, 15 Apr 2026 21:02:02 GMT
• Title: Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality
• Abstract summary: We build a sparse Mixture-of-Experts (MoE) architecture using cosine-similarity routing against learned centroids.<n>We find that routing topology does not determine perplexity (PPL)<n>Expert-level and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sparse Mixture-of-Experts (MoE) architectures employ increasingly sophisticated routing mechanisms -- learned routers, multi-hop trajectories, token-dependent gating. We ask: does routing topology actually determine language modeling quality? We build a geometric MoE (ST-MoE) using cosine-similarity routing against learned centroids in a low-dimensional space ($d_{space} = 64$), requiring 80% fewer routing parameters than standard linear routers. Through 62 controlled experiments on WikiText-103 at 76--84M parameters trained to convergence (50K steps, 1.64B tokens), we find that routing topology does not determine asymptotic perplexity (PPL): five cosine-routing variants are statistically equivalent within a 1-PPL margin (Two One-Sided Tests [TOST], $p < 0.05$ for all 10 pairwise comparisons; 15 runs across 3 seeds, observed range 33.93--34.72). The finding extends to hash, random-fixed, and top-1 routing (single-seed; graceful 1.1--2.2 PPL degradation) and replicates on OpenWebText (0.03 PPL gap, 6 runs, 3 seeds each). A standard linear router with 5.3$\times$ more routing parameters reaches PPL 32.76, but iso-parameter cosine routing closes 67% of this gap -- the true mechanism advantage is $\sim$1.2%. The mechanistic explanation is convergent redundancy: multi-hop updates are collinear ($\cos(Δh_0, Δh_1) = 0.805$), implementing magnitude amplification rather than compositional reasoning; a single learnable scalar replicates multi-hop performance. As a practical payoff, zero-shot relative-norm halting saves 25% of MoE FLOPs at +0.12% PPL. Expert-level specialization and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.

Related papers

• HetRoute Heterogeneous and Cost-aware Collaborative Routing Framework for Distributed Edge MoE Inference [47.0879800334801]
  HetRoute is a heterogeneous-cost-aware collaborative routing framework for distributed edge MoE inference.<n>HetRoute reduces average inference latency by up to 59.0% and P99 latency by up to 58.0%.
  arXiv  Detail & Related papers  (2026-08-01T10:30:01Z)
• CLEAR-MoE: Shared-Basis Expert Extraction from Frozen Vision Transformers via Calibration-Driven Layer Selection [0.0]
  CLEAR-MoE is a post-training pipeline that converts a frozen pretrained Vision Transformer (ViT) into a sparse Mixture-of-Experts (MoE) model without updating backbone weights.<n>On a GTX 960, routing and scatter-gather overhead make the CLEAR-MoE FFN 1.3-1.7x slower than the dense routers.
  arXiv  Detail & Related papers  (2026-06-26T18:12:03Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• LayerRoute: Input-Conditioned Adaptive Layer Skipping via LoRA Fine-Tuning for Agentic Language Models [0.0]
  We introduce LayerRoute, a lightweight adapter that learns to selectively skip transformer blocks on a per-input basis.<n> LayerRoute augments each of the 24 transformer blocks in Qwen2.5-0.5B-Instruct with: (1) a per-layer router that outputs a hard binary gate via the straight-through estimator, and (2) LoRA adapters (rank 8, 1.08M parameters) on the Q/K/V/O attention projections.<n>A single end-to-end training pass on agentic data (Hermes, Glaive, GSM8K, Turing) with a gate regularisation term
  arXiv  Detail & Related papers  (2026-06-01T07:50:05Z)
• CART: Context-Anchored Recurrent Transformer -- A Parameter-Efficient Architecture with Learned Stability [0.0]
  CART (Context-Anchored Recurrent Transformer) is a parameter-efficient language model that reuses a single shared core block R times across depth.<n>We evaluate CART on single consumer GPUs in two stages: a 64-configuration screen at 3,000 steps, then 36 configurations (P=6, R in 6,8,10, three seeds) trained for 30,500 steps (1B tokens)<n>Two patterns hold across widths d in 256,512,768,1024: prelude depth P dominates loop count R, and the Stage-1 ranking of R reverses at full training (R=6 becomes best at
  arXiv  Detail & Related papers  (2026-05-31T23:26:27Z)
• HubRouter: A Pluggable Sub-Quadratic Routing Primitive for Hybrid Sequence Models [0.0]
  We introduce Hub Code, a pluggable module that replaces O(n2) attention layers with O(nM) hub-mediated routing.<n>We demonstrate it in two from-scratch architectures: a Jamba-style hybrid and a 12-layer Transformer.
  arXiv  Detail & Related papers  (2026-04-24T10:59:30Z)
• Regime-Conditional Retrieval: Theory and a Transferable Router for Two-Hop QA [0.0]
  Two-hop QA retrieval queries splits into two regimes determined by whether the hop-2 entity is explicitly named in the question (Q-dominant) or only in the bridge passage (B-dominant)<n>We formalize this split with three theorems.<n>Building on this theory, we propose a lightweight binary router that selects between question-only and question-plusrelation-sentence retrieval.
  arXiv  Detail & Related papers  (2026-04-10T06:31:51Z)
• Coupled Query-Key Dynamics for Attention [6.775853253396773]
  We show evolving queries and keys emphjointly through shared learned dynamics before scoring.<n>On WikiText-103 at 60M parameters, coupled dynamics achieves 22.55--22.62 perplexity vs. 24.22 for standard attention.
  arXiv  Detail & Related papers  (2026-04-02T06:37:05Z)
• KALAVAI: Predicting When Independent Specialist Fusion Works -- A Quantitative Model for Post-Hoc Cooperative LLM Training [3.151184728006369]
  Independently trained domain specialists can be fused post-hoc into a single model that outperforms any individual specialist.<n>In the KALAVAI protocol, contributors fine-tune copies of a shared checkpoint independently, then submit for lightweight MoE routing (500 steps)
  arXiv  Detail & Related papers  (2026-03-24T03:32:04Z)
• Directional Routing in Transformers [0.0]
  directional routing is a lightweight mechanism that gives each transformer attention head learned suppression directions controlled by a shared router.<n>We trace the resulting circuits through mechanistic interpretability.<n> routing reduces perplexity 31-56% relative to the baseline, though downstream multiple-choice benchmarks do not yet reflect these gains.
  arXiv  Detail & Related papers  (2026-03-16T07:28:22Z)
• Finding Transformer Circuits with Edge Pruning [71.12127707678961]
  We propose Edge Pruning as an effective and scalable solution to automated circuit discovery.<n>Our method finds circuits in GPT-2 that use less than half the number of edges compared to circuits found by previous methods.<n>Thanks to its efficiency, we scale Edge Pruning to CodeLlama-13B, a model over 100x the scale that prior methods operate on.
  arXiv  Detail & Related papers  (2024-06-24T16:40:54Z)
• Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks [74.68583356645276]
  In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis. We show for the first time that pMoE provably reduces the required number of training samples to achieve desirable generalization.
  arXiv  Detail & Related papers  (2023-06-07T00:16:10Z)
• Retrosynthetic Planning with Dual Value Networks [107.97218669277913]
  We propose a novel online training algorithm, called Planning with Dual Value Networks (PDVN) PDVN alternates between the planning phase and updating phase to predict the synthesizability and cost of molecules. On the widely-used USPTO dataset, our PDVN algorithm improves the search success rate of existing multi-step planners.
  arXiv  Detail & Related papers  (2023-01-31T16:43:53Z)
• Learning to Generate Content-Aware Dynamic Detectors [62.74209921174237]
  We introduce a newpective of designing efficient detectors, which is automatically generating sample-adaptive model architecture. We introduce a course-to-fine strat-egy tailored for object detection to guide the learning of dynamic routing. Experiments on MS-COCO dataset demonstrate that CADDet achieves 1.8 higher mAP with 10% fewer FLOPs compared with vanilla routing.
  arXiv  Detail & Related papers  (2020-12-08T08:05:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.