Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts

• URL: http://arxiv.org/abs/2605.12476v1
• Date: Tue, 12 May 2026 17:55:02 GMT
• Title: Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
• Abstract summary: We study how routing decisions in SMoEs are formed mechanistically.<n>In a $1$B SMoE trained from scratch, higher router scores predict stronger expert neuron activations.<n>Our results explain how routers form assignment geometry that supports an effective division of labor.
• Score: 23.123497435269112
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse Mixture-of-Experts (SMoE) models enable scaling language models efficiently, but training them remains challenging, as routing can collapse onto few experts and auxiliary load-balancing losses can reduce specialization. Motivated by these hurdles, we study how routing decisions in SMoEs are formed mechanistically. First, we reveal a geometric coupling between routers and their corresponding experts. For a given token, the router weights for the selected expert and the expert weights processing it receive gradients along the same input direction, differing only in scalar coefficients. Thus, matched router--expert directions accumulate the same routed token history. This theoretical coupling also appears empirically in routing dynamics. In a $1$B SMoE trained from scratch, higher router scores predict stronger expert neuron activations, showing that routing decisions are mirrored inside the selected expert. Next, we analyze the effects of auxiliary load balancing on the router--expert geometric coupling, showing that such losses break this structure by spreading input-directed gradients across router weights, making distinct router directions nearly three times more similar to each other. Last, we demonstrate the centrality of geometric coupling for effective routing with a parameter-free online K-Means router, in which each expert maintains a running average of the hidden states routed to it and tokens are assigned based on cosine similarity. Compared with auxiliary-loss and loss-free balancing, this router achieves the lowest load imbalance with only a modest perplexity increase, indicating that geometric coupling captures a substantial part of what the router learns. Overall, our results explain how routers form assignment geometry that supports an effective division of labor.

Related papers

• Share First, Route What Remains: A Unified Framework for Token-Adaptive MoE Computation [49.8459545291965]
  Mixture-of-experts (MoE) models have recently moved beyond routing a fixed number of complete experts.<n>We study this dependency by decomposing sparsely upcycled feed-forward experts into key-value channels.<n>We instantiate it in UniF-MoE, a unified framework for token-adaptive MoE computation.
  arXiv  Detail & Related papers  (2026-08-11T02:40:58Z)
• Redesign Mixture-of-Experts Routers with Manifold Power Iteration [60.55368504665436]
  We propose a router redesign with Manifold Power Iteration (MPI)<n>MPI drives router rows to converge toward the principal singular directions of associated experts.<n>We show that MPI drives router rows to converge toward the principal singular directions of associated experts.
  arXiv  Detail & Related papers  (2026-06-10T17:57:36Z)
• Adaptive Inverted-Index Routing for Granular Mixtures-of-Experts [49.09151538536423]
  Mixture-of-experts (MoE) models enable scalable transformer architectures by activating only a subset of experts per token.<n>Recent evidence suggests that performance improves with increasingly granular experts, i.e., many small experts instead of a few large ones.<n>We introduce adaptive inverted-index routing for MoE (AIR-MoE), an inverted-index-inspired routing architecture based on vector quantization (VQ)
  arXiv  Detail & Related papers  (2026-05-06T14:15:10Z)
• Self-Routing: Parameter-Free Expert Routing from Hidden States [14.018297453424454]
  Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token.<n>We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits.
  arXiv  Detail & Related papers  (2026-04-01T03:05:20Z)
• ReMix: Reinforcement routing for mixtures of LoRAs in LLM finetuning [85.39146836671897]
  Low-rank adapters (LoRAs) are a parameter-efficient finetuning technique that injects trainable low-rank matrices into pretrained models to adapt them to new tasks.<n>Existing Mixture-of-LoRAs routers assign a learned routing weight to each LoRA to enable end-to-end training of the router.<n>We propose a new router designed that we call Reinforcement Routing for Mixture-of-LoRAs (ReMix)
  arXiv  Detail & Related papers  (2026-03-10T18:51:27Z)
• L2R: Low-Rank and Lipschitz-Controlled Routing for Mixture-of-Experts [49.90176890917986]
  We propose Low-rank & Lipschitz-controlled Routing (L2R), a unified routing framework that reshapes both the routing space and scoring geometry.<n>L2R performs expert assignment in a shared low-rank latent routing space and introduces Saturated Inner-Product Scoring (SIPS) to explicitly control the Lipschitz behavior of routing functions.<n>Experiments on a large-scale language MoE model and a vision MoE setting on ImageNet demonstrate that L2R consistently improves routing stability, expert specialization, and overall model performance.
  arXiv  Detail & Related papers  (2026-01-29T07:18:33Z)
• Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts [32.65737144630759]
  Mixture-of-Experts (MoE) architectures scale large language models efficiently by employing a parametric "router" to dispatch tokens to a sparse subset of experts.<n>We introduce kNN-MoE, a retrieval-augmented routing framework that reuses optimal expert assignments from a memory of similar past cases.<n>Experiments show kNN-MoE outperforms zero-shot baselines and rivals computationally expensive supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-01-05T14:16:11Z)
• AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering [51.07491603393163]
  tAgent is a framework that formulates multi-agent QA as a knowledge-graph-guided routing problem supervised by empirical performance signals.<n>By leveraging soft supervision and weighted aggregation of agent outputs, Agent learns principled collaboration schemes that capture the complementary strengths of diverse agents.
  arXiv  Detail & Related papers  (2025-10-06T23:20:49Z)
• From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing [52.01745035243826]
  Mixture-of-Experts (MoE) models can scale parameter capacity by routing each token to a subset of experts.<n> conditional routing shifts the burden on inference memory, limiting the number of experts per device.<n>We present LASER, a plug-and-play, inference-time routing algorithm that balances load while preserving accuracy.
  arXiv  Detail & Related papers  (2025-09-29T16:29:17Z)
• Load Balancing Mixture of Experts with Similarity Preserving Routers [30.279616888339543]
  Sparse Mixture of Experts (MoE) models offer a scalable and efficient architecture for training large neural networks.<n>We introduce a novel load balancing loss that preserves token-wise relational structure.<n>Our results show that applying our loss to the router results in 36% faster convergence and lower redundancy.
  arXiv  Detail & Related papers  (2025-06-16T22:22:59Z)
• HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts [34.08858035082419]
  This work introduces HyperRout, which dynamically generates the router's parameters through a fixed hypernetwork and trainable embeddings. Experiments across a wide range of tasks demonstrate the superior performance and efficiency gains of HyperRout.
  arXiv  Detail & Related papers  (2023-12-12T07:40:23Z)
• Towards Bi-directional Skip Connections in Encoder-Decoder Architectures and Beyond [95.46272735589648]
  We propose backward skip connections that bring decoded features back to the encoder. Our design can be jointly adopted with forward skip connections in any encoder-decoder architecture. We propose a novel two-phase Neural Architecture Search (NAS) algorithm, namely BiX-NAS, to search for the best multi-scale skip connections.
  arXiv  Detail & Related papers  (2022-03-11T01:38:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.