Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts

• URL: http://arxiv.org/abs/2609.02404v1
• Date: Wed, 02 Sep 2026 10:17:12 GMT
• Title: Evidence for Shared Routing Geometry and Dynamics in Sparse Mixture-of-Experts
• Abstract summary: We show that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems.<n>A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sparse mixture-of-experts (MoE) models use an independently parameterized router at each sparse layer to select experts for every token. Prior work has shown that routing decisions across depth can often be predicted from earlier routing signals, suggesting that routing is not fully independent across layers. However, the structure behind this predictability remains unclear. In this work, we provide evidence that routing-relevant states across layers share a common geometric structure that is obscured by layer-specific coordinate systems. We isolate the control subspace of each router and align these spaces into a shared canonical representation using generalized orthogonal Procrustes analysis. After alignment, a single linear transition reaches $R^2=0.39$--$0.71$ and retains 79--90\% of the predictive power of separately fitted layer-specific dynamics, indicating that much of routing-state evolution follows a reusable process across depth. We then ask whether this shared dynamics is specific to routing or simply reflects the smooth evolution of hidden representations. A matched-rank comparison shows that residual representations are often easier to predict across layers, while router-control states preserve the model's expert choices much more faithfully. This separates generic cross-layer predictability from routing-specific information. Finally, we test whether the predicted canonical states remain meaningful when used in place of native routing states. The transported states preserve local routing behavior, while learned state evolution reduces $Δ\mathrm{NLL}$ relative to simple persistence by 15.7\% on OLMoE and 6.2\% over a 10-router horizon on Phi.

Related papers

• Do Dynamic Routers Need Memory? HeRo: History-Aware Routing for Efficient LLM Inference [13.935681015472262]
  Dynamic layer routing reduces the inference cost of Large Language Models (LLMs) by learning to skip layers for individual tokens.<n>Existing methods treat each routing decision as a local operation conditioned solely on the current hidden state.<n>We propose History-Aware Routing (HeRo), a dynamic routing framework that maintains an explicit routing state across model depth.
  arXiv  Detail & Related papers  (2026-09-08T03:20:07Z)
• Disagree to Explore, Agree to Commit: Routing-Guided Test-Time Scaling for Software Agents [60.650808962786364]
  We study whether native MoE router traces can guide steering and selection without an external judge or selection-time test execution.<n>Our analysis shows that routing provides a robust behavioral role signal; token-granular readouts and decision-matched comparison sets turn it into effective control.<n>We evaluate on SWE-bench Verified using open-weight sparse MoE agents across scales and reasoning-effort settings.
  arXiv  Detail & Related papers  (2026-08-23T03:07:03Z)
• Self-Routing: Parameter-Free Expert Routing from Hidden States [14.018297453424454]
  Mixture-of-Experts (MoE) layers increase model capacity by activating only a small subset of experts per token.<n>We propose Self-Routing, a parameter-free routing mechanism that uses a designated subspace of the token hidden state directly as expert logits.
  arXiv  Detail & Related papers  (2026-04-01T03:05:20Z)
• Path-Constrained Mixture-of-Experts [23.80197713200086]
  MoE routing selects each layer's experts independently, creating NL possible expert paths -- for N experts across L layers.<n>We propose pathmoe, which shares router parameters across consecutive layers.<n>Experiments on 0.9B and 16B parameter models demonstrate consistent improvements on perplexity and downstream tasks over independent routing.
  arXiv  Detail & Related papers  (2026-03-18T21:35:53Z)
• Task-Conditioned Routing Signatures in Sparse Mixture-of-Experts Transformers [0.0]
  We introduce routing signatures, a vector representation summarizing expert activation patterns across layers for a given prompt.<n>We show that prompts from the same task category induce highly similar routing signatures, while prompts from different categories exhibit substantially lower similarity.<n>A logistic regression trained solely on routing signatures achieves 92.5% +/- 6.1% cross-validated accuracy on four-way task classification.
  arXiv  Detail & Related papers  (2026-03-11T12:45:53Z)
• RealRoute: Dynamic Query Routing System via Retrieve-then-Verify Paradigm [19.239852494559212]
  We introduce RealRoute, a framework that shifts the paradigm from predictive routing to a robust Retrieve-then-Verify mechanism.<n> Experiments show that RealRoute significantly outperforms predictive baselines in the multi-hop Rag reasoning task.
  arXiv  Detail & Related papers  (2026-03-02T17:57:46Z)
• Unifying Environment Perception and Route Choice Modeling for Trajectory Representation Learning [47.00223863430964]
  Tray Learning (TRL) aims to encode raw trajectories into low-dimensional vectors, which can be leveraged in various downstream tasks, including travel time estimation, location prediction, and trajectory similarity analysis.<n>We propose a framework that unifies comprehensive environment textbfPertemporal explicit textRoute choice modeling for effective textbfPRTrajectory representation learning, dubbed textbfPRTraj.
  arXiv  Detail & Related papers  (2025-10-16T15:55:28Z)
• Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition [22.046185271408202]
  Mixture-of-experts (MoE) architectures have expanded from language modeling to automatic speech recognition (ASR)<n>Our analysis reveals that routers in most layers make expert choices that are not strongly correlated with the choices of the routers in other layers.<n>To increase the cooperation between experts in different layers and encourage greater specialization, we use a shared router across different MoE layers.
  arXiv  Detail & Related papers  (2025-07-08T07:18:33Z)
• Find A Winning Sign: Sign Is All We Need to Win the Lottery [52.63674911541416]
  We show that a sparse network trained by an existing IP method can retain its basin of attraction if its parameter signs and normalization layer parameters are preserved.<n>To take a step closer to finding a winning ticket, we alleviate the reliance on normalization layer parameters by preventing high error barriers along the linear path between the sparse network trained by our method and its counterpart with normalization layer parameters.
  arXiv  Detail & Related papers  (2025-04-07T09:30:38Z)
• Learning to Generate Content-Aware Dynamic Detectors [62.74209921174237]
  We introduce a newpective of designing efficient detectors, which is automatically generating sample-adaptive model architecture. We introduce a course-to-fine strat-egy tailored for object detection to guide the learning of dynamic routing. Experiments on MS-COCO dataset demonstrate that CADDet achieves 1.8 higher mAP with 10% fewer FLOPs compared with vanilla routing.
  arXiv  Detail & Related papers  (2020-12-08T08:05:20Z)
• DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
  This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
  arXiv  Detail & Related papers  (2020-03-30T17:59:18Z)
• Learning Dynamic Routing for Semantic Segmentation [86.56049245100084]
  This paper studies a conceptually new method to alleviate the scale variance in semantic representation, named dynamic routing. The proposed framework generates data-dependent routes, adapting to the scale distribution of each image. To this end, a differentiable gating function, called soft conditional gate, is proposed to select scale transform paths on the fly.
  arXiv  Detail & Related papers  (2020-03-23T17:22:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.