Redesign Mixture-of-Experts Routers with Manifold Power Iteration

• URL: http://arxiv.org/abs/2606.12397v1
• Date: Wed, 10 Jun 2026 17:57:36 GMT
• Title: Redesign Mixture-of-Experts Routers with Manifold Power Iteration
• Abstract summary: We propose a router redesign with Manifold Power Iteration (MPI)<n>MPI drives router rows to converge toward the principal singular directions of associated experts.<n>We show that MPI drives router rows to converge toward the principal singular directions of associated experts.
• Score: 60.55368504665436
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Router is the cornerstone component to the Mixture-of-Experts models. Serving as expert proxies, the rows of the router matrix compute their similarity to the MoE inputs to determine which subset of experts is activated. Ideally, each router row is designed to encode the expert matrix into this representative vector, such that its dot-product with token can better reflect token-expert affinity. However, there exists no design principles to enforce this condensation. In this paper, we propose to align each router row with the principal singular direction of the associated expert, as this direction provides the most expressive mathematical description of a matrix. Based on this principle, we propose a router redesign with Manifold Power Iteration (MPI). Specifically, it introduces a "Power-then-Retract" paradigm, where a power iteration step is performed on the router weights, followed by a retraction to impose a norm constraint to ensure both efficiency and stability. Theoretically, we show that MPI drives router rows to converge toward the principal singular directions of associated experts. Empirically, we pretrain MoE model across scales from 1B to 11B parameters to confirm that this alignment facilitates more effective MoE models.

Related papers

• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts [23.123497435269112]
  We study how routing decisions in SMoEs are formed mechanistically.<n>In a $1$B SMoE trained from scratch, higher router scores predict stronger expert neuron activations.<n>Our results explain how routers form assignment geometry that supports an effective division of labor.
  arXiv  Detail & Related papers  (2026-05-12T17:55:02Z)
• Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts [49.09151538536423]
  Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token.<n>Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones.<n>We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ)
  arXiv  Detail & Related papers  (2026-05-06T14:15:10Z)
• Self-Routing: Parameter-Free Expert Routing from Hidden States [14.018297453424454]
  Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token.<n>We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits.
  arXiv  Detail & Related papers  (2026-04-01T03:05:20Z)
• Least-Loaded Expert Parallelism: Load Balancing An Imbalanced Mixture-of-Experts [74.40169987564724]
  Expert parallelism (EP) is designed to scale MoE models by distributing experts across multiple devices.<n>Under extreme imbalance, EP can funnel a disproportionate number of tokens to a small number of experts, leading to compute- and memory-bound failures.<n>We propose Least-Loaded Expert Parallelism (LLEP), a novel EP algorithm that dynamically reroutes excess tokens and associated expert parameters from overloaded devices to underutilized ones.
  arXiv  Detail & Related papers  (2026-01-23T18:19:15Z)
• Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts [32.65737144630759]
  Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric "router" to dispatch tokens to a sparse subset of experts.<n>We introduce kNN-MoE, a retrieval-augmented routing framework that reuses optimal expert assignments from a memory of similar past cases.<n>Experiments show kNN-MoE outperforms zero-shot baselines and rivals computationally expensive supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-01-05T14:16:11Z)
• Routing Matters in MoE: Scaling Diffusion Transformers with Explicit Routing Guidance [79.21541758879012]
  Mixture-of-Experts (MoE) has emerged as a powerful paradigm for scaling model capacity while preserving computational efficiency.<n>We present ProMoE, an MoE framework featuring a two-step router with explicit routing guidance that promotes expert specialization.
  arXiv  Detail & Related papers  (2025-10-28T17:59:02Z)
• Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel [87.60286115014833]
  Mixture-of-Experts (MoE) has become a cornerstone in recent state-of-the-art large language models (LLMs)<n>Traditionally, MoE relies on $mathrmSoftmax$ as the router score function to aggregate expert output.<n>We propose the textbfzero-additional-cost Kernel Router with Normalization (KERN) as an alternative to $mathrmSoftmax$.
  arXiv  Detail & Related papers  (2025-09-30T08:04:02Z)
• Optimizing MoE Routers: Design, Implementation, and Evaluation in Transformer Models [0.0]
  Mixture of Experts (MoE) architectures increase large language model scalability, yet their performance depends on the router module that moves tokens to specialized experts.<n>This work provides a comparative analysis of MoE router designs and offers insights into optimizing their performance for efficient and effective large-scale model deployment.
  arXiv  Detail & Related papers  (2025-06-19T15:55:43Z)
• Routers in Vision Mixture of Experts: An Empirical Study [26.51711534240885]
  Mixture-of-Experts (MoE) models are a promising way to scale up model capacity without significantly increasing computational cost. Key component of MoEs is the router, which decides which subset of parameters (experts) process which feature embeddings (tokens)
  arXiv  Detail & Related papers  (2024-01-29T08:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.