Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts

• URL: http://arxiv.org/abs/2605.20610v1
• Date: Wed, 20 May 2026 01:55:40 GMT
• Title: Beyond Routing: Characterising Expert Tuning and Representation in Vision Mixture-of-Experts
• Abstract summary: Mixture-of-Experts (MoE) models are often interpreted by analysing categories which are routed to which experts.<n>We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and expert specialisation.<n>We measure per-expert category separability, and per-expert tuning using the most exciting inputs.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models are often interpreted by analysing which categories are routed to which experts. However, routing alone does not reveal what each expert actually encodes. We train sparsely-gated convolutional MoE models with a contrastive objective on natural images and characterise expert specialisation using tools from visual neuroscience. Extending from gating-level to expert-level analyses, we measure per-expert category separability, and per-expert tuning using the most exciting inputs. Extending from category-level to feature-level explanations, we interpret tuning via semantic dimensions derived from a dataset of human behavioural judgements (THINGS). Finally, we use tuning and representational similarity analysis to assess the stability of expertise-allocation across independent initialisations. We find that an animate-inanimate distinction dominates expert partitioning, apparent from gating through to expert readout, and is stable across independently trained models. Although routing statistics suggest relatively sparse, categorical preferences, expert analyses reveal broader tuning to continuous visual and semantic dimensions that extend beyond category boundaries. Experts exhibit similar category-separability to one another, despite distinct feature tuning, demonstrating the explanatory benefits of moving beyond category-level analyses. Together, these results show that expert specialisation in vision MoEs extends well beyond category routing and is better understood by probing fine-grained expert-level tuning and representational structure.

Related papers

• Slicing and Dicing: Configuring Optimal Mixtures of Experts [49.5613403644084]
  We present the first systematic study of over 2,000 pretraining runs spanning models up to 6.6B total parameters.<n>We exhaustively vary total experts, expert dimension, heterogeneous expert sizing within a single layer, shared expert size and load-balancing mechanisms.<n>Our results suggest a simpler recipe: focus on expert count and granularity, other choices have minimal effect on final quality.
  arXiv  Detail & Related papers  (2026-05-12T07:47:22Z)
• DeferredSeg: A Multi-Expert Deferral Framework for Trustworthy Medical Image Segmentation [66.24948946364329]
  We introduce DeferredSeg, a deferral-aware segmentation framework that defers predictions to human experts in specific regions.<n>We evaluate DeferredSeg on three challenging medical datasets using MedSAM and CENet as the base segmentor for fair comparison.<n> Experimental results show that DeferredSeg consistently outperforms the baseline.
  arXiv  Detail & Related papers  (2026-04-14T07:50:31Z)
• MoE Lens -- An Expert Is All You Need [0.09444932939944471]
  Mixture of Experts (MoE) models enable parameter-efficient scaling through sparse expert activations.<n>We present a systematic analysis of expert specialization in MoEs through two complementary approaches.
  arXiv  Detail & Related papers  (2026-03-06T01:35:28Z)
• The Illusion of Specialization: Unveiling the Domain-Invariant "Standing Committee" in Mixture-of-Experts Models [18.428606280260187]
  Mixture of Experts models are widely assumed to achieve domain specialization through sparse routing.<n>We introduce COMMITTEEAUDIT, a framework that analyzes routing behavior at the level of expert groups rather than individual experts.<n>We find that Standing Committees consistently capture the majority of routing mass across domains, layers, and routing budgets.
  arXiv  Detail & Related papers  (2026-01-06T21:29:45Z)
• Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations [48.890534958441016]
  This study investigates domain specialization and expert redundancy in large-scale MoE models.<n>We propose a simple yet effective pruning framework, EASY-EP, to identify and retain only the most relevant experts.<n>Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99times$ throughput under the same memory budget with full model with only half the experts.
  arXiv  Detail & Related papers  (2025-04-09T11:34:06Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study [65.11303133775857]
  Mixture-of-Experts (MoE) computation amalgamates predictions from several specialized sub-models (referred to as experts) Sparse MoE selectively engages only a limited number, or even just one expert, significantly reducing overhead while empirically preserving, and sometimes even enhancing, performance.
  arXiv  Detail & Related papers  (2024-03-26T05:48:02Z)
• Sparsely-gated Mixture-of-Expert Layers for CNN Interpretability [3.021134753248103]
  Sparsely-gated Mixture of Expert (MoE) layers have been successfully applied for scaling large transformers. In this work, we apply sparse MoE layers to CNNs for computer vision tasks and analyze the resulting effect on model interpretability.
  arXiv  Detail & Related papers  (2022-04-22T09:40:23Z)
• On the Representation Collapse of Sparse Mixture of Experts [102.83396489230375]
  Sparse mixture of experts provides larger model capacity while requiring a constant computational overhead. It employs the routing mechanism to distribute input tokens to the best-matched experts according to their hidden representations. However, learning such a routing mechanism encourages token clustering around expert centroids, implying a trend toward representation collapse.
  arXiv  Detail & Related papers  (2022-04-20T01:40:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.