Self-Routing: Parameter-Free Expert Routing from Hidden States

• URL: http://arxiv.org/abs/2604.00421v1
• Date: Wed, 01 Apr 2026 03:05:20 GMT
• Title: Self-Routing: Parameter-Free Expert Routing from Hidden States
• Abstract summary: Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token.<n>We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits.
• Score: 14.018297453424454
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token, and typically rely on a learned router to map hidden states to expert assignments. In this work, we ask whether a dedicated learned router is strictly necessary in the MoE settings we study. We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits, eliminating the router projection entirely while leaving the rest of the MoE layer unchanged. We evaluate Self-Routing on GPT-2-scale language modeling and ImageNet-1K classification by comparing it against a standard learned router, random-routing baselines, and dense non-MoE baselines. Our results show that Self-Routing remains competitive with the learned-router baseline while removing all dedicated routing parameters, and yields more balanced expert utilization, with about 17 % higher average normalized routing entropy and no explicit load-balancing loss. On ImageNet-1K with DeiT-S/16, Self-Routing also slightly improves over the corresponding learned-router MoE. These findings suggest that effective MoE routing can emerge from the hidden representation itself without requiring a separate learned router module.

Related papers

• Agent-as-a-Router: Agentic Model Routing for Coding Tasks [54.00153674547733]
  Agent-as-a-router (AC) formalizes routing as a C-A-F loop (Context->Action->FeedbackContext)<n>AC achieves the lowest cumulative regret on in-distribution tasks and generalizes to out-of-distribution agentic-programming tasks.
  arXiv  Detail & Related papers  (2026-06-22T06:37:31Z)
• Redesign Mixture-of-Experts Routers with Manifold Power Iteration [60.55368504665436]
  We propose a router redesign with Manifold Power Iteration (MPI)<n>MPI drives router rows to converge toward the principal singular directions of associated experts.<n>We show that MPI drives router rows to converge toward the principal singular directions of associated experts.
  arXiv  Detail & Related papers  (2026-06-10T17:57:36Z)
• Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts [23.123497435269112]
  We study how routing decisions in SMoEs are formed mechanistically.<n>In a $1$B SMoE trained from scratch, higher router scores predict stronger expert neuron activations.<n>Our results explain how routers form assignment geometry that supports an effective division of labor.
  arXiv  Detail & Related papers  (2026-05-12T17:55:02Z)
• When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models [27.942734943134983]
  Mixture-of-Experts (MoE) language models route each token to a small subset of experts.<n>We compare each standard route against sampled equal-compute alternatives for the same token and score each by the next-token probability it assigns to the realized token in a verified reasoning trajectory.<n>The result is sharply token-conditional: the standard router is well-aligned with route utility on confident tokens but uninformative on the fragile tokens that drive hard reasoning.
  arXiv  Detail & Related papers  (2026-05-08T05:26:09Z)
• Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts [49.09151538536423]
  Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token.<n>Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones.<n>We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ)
  arXiv  Detail & Related papers  (2026-05-06T14:15:10Z)
• ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning [85.39146836671897]
  Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks.<n>Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router.<n>We propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix)
  arXiv  Detail & Related papers  (2026-03-10T18:51:27Z)
• L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts [49.90176890917986]
  We propose Low-rank & Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry.<n>L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions.<n>Experiments on a large-scale language MoE model and a vision MoE setting on ImageNet demonstrate that L2R consistently improves routing stability, expert specialization, and overall model performance.
  arXiv  Detail & Related papers  (2026-01-29T07:18:33Z)
• Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts [32.65737144630759]
  Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric "router" to dispatch tokens to a sparse subset of experts.<n>We introduce kNN-MoE, a retrieval-augmented routing framework that reuses optimal expert assignments from a memory of similar past cases.<n>Experiments show kNN-MoE outperforms zero-shot baselines and rivals computationally expensive supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-01-05T14:16:11Z)
• From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing [52.01745035243826]
  Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts.<n> conditional routing shifts the burden on inference memory, limiting the number of experts per device.<n>We present LASER, a plug-and-play, inference-time routing algorithm that balances load while preserving accuracy.
  arXiv  Detail & Related papers  (2025-09-29T16:29:17Z)
• Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models [0.0]
  Mixture of Experts (MoE) architectures increase large language model scalability, yet their performance depends on the router module that moves tokens to specialized experts.<n>This work provides a comparative analysis of MoE router designs and offers insights into optimizing their performance for efficient and effective large-scale model deployment.
  arXiv  Detail & Related papers  (2025-06-19T15:55:43Z)
• Universal Model Routing for Efficient LLM Inference [69.86195589350264]
  Model routing is a technique for reducing the inference cost of large language models (LLMs)<n>We propose UniRoute, a new approach to the problem of dynamic routing, where new, previously unobserved LLMs are available at test time.<n>We show that these are estimates of a theoretically optimal routing rule, and quantify their errors via an excess risk bound.
  arXiv  Detail & Related papers  (2025-02-12T20:30:28Z)
• ReMoE: Fully Differentiable Mixture-of-Experts with ReLU Routing [28.73697327316267]
  Sparsely activated Mixture-of-Experts (MoE) models are widely adopted to scale up model capacity without increasing the computation budget.<n>We propose ReMoE, a fully differentiable MoE architecture that offers a simple yet effective drop-in replacement for the conventional TopK+Softmax routing.<n>ReMoE consistently outperforms vanilla TopK-routed MoE across various model sizes, expert counts, and levels of granularity.
  arXiv  Detail & Related papers  (2024-12-19T10:21:20Z)
• Routers in Vision Mixture of Experts: An Empirical Study [26.51711534240885]
  Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. Key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embeddings (tokens)
  arXiv  Detail & Related papers  (2024-01-29T08:58:07Z)
• Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks [74.68583356645276]
  In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis. We show for the first time that pMoE provably reduces the required number of training samples to achieve desirable generalization.
  arXiv  Detail & Related papers  (2023-06-07T00:16:10Z)
• StableMoE: Stable Routing Strategy for Mixture of Experts [109.0602120199226]
  Mixture-of-Experts (MoE) technique can scale up the model size of Transformers with an affordable computational overhead. We propose StableMoE with two training stages to address the routing fluctuation problem. Results show that StableMoE outperforms existing MoE methods in terms of both convergence speed and performance.
  arXiv  Detail & Related papers  (2022-04-18T16:48:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.