When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing

• URL: http://arxiv.org/abs/2605.15484v1
• Date: Fri, 15 May 2026 00:01:11 GMT
• Title: When Does Sparse MoE Help in Vision? The Role of Backbone Compute Leverage in Sparse Routing
• Abstract summary: Mixture-of-Experts (MoE) networks promise favorable accuracy-compute trade-offs, yet practical vision deployments are hindered by expert collapse.<n>We study when sparse top-$k$ routing with hard capacity constraints helps in vision classification.
• Score: 7.208745673318648
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) networks promise favorable accuracy-compute trade-offs, yet practical vision deployments are hindered by expert collapse and limited end-to-end efficiency gains. We study when sparse top-$k$ routing with hard capacity constraints helps in vision classification, evaluated under multi-seed protocols on four benchmarks (CIFAR-10/100, Tiny-ImageNet, ImageNet-1K). We observe a \emph{compute-leverage pattern}: positive accuracy gaps require a substantial fraction $ρ$ of total FLOPs to be routed; at ImageNet scale this is necessary but not sufficient, as multi-expert routing ($k \geq 2$) is additionally required. Two controlled experiments isolate these factors. A hidden-size sweep on CIFAR-10 yields both predicted sign reversals across standard and depthwise backbones, ruling out backbone family as the active variable. An ImageNet-1K ablation that varies only top-$k$ -- holding architecture, initialization, and $ρ$ fixed -- reverses the gap from positive to negative across all five seeds. A per-sample variant of Soft MoE that softmaxes over experts rather than the batch rescues CIFAR-100 above the dense baseline, identifying batch-axis dispatch as the dominant failure mode in per-sample CNN settings. Code and aggregate results: https://github.com/libophd/sparse-moe-vision-rho.

Related papers

• Classical $\mathrm{SU}(2)$ Models Match or Exceed Shallow Variational Quantum Circuits on Vision Benchmarks [0.0]
  Quaternion-valued neural networks and variational quantum circuits (VQCs) both derive local transformations from $mathrmSU(2)$ geometry.<n>We compare real-valued, quaternionvalued, and quantum classification heads on identical frozen features across MNIST, FashionMNIST, and CIFAR-10.<n>Our results indicate that quaternion networks provide efficient, stable alternatives to shallow VQCs on tasks lacking intrinsic quantum structure.
  arXiv  Detail & Related papers  (2026-08-07T23:50:09Z)
• SHAPE: Coalition-Aware Expert Pruning for Sparse Mixture-of-Experts LLMs [4.035485260002141]
  Sparse Mixture-of-Experts (MoE) models achieve strong quality with low per-token compute, yet their deployment is often limited by the memory wall.<n>We propose textbfSHAPE, a task-driven pruning framework that explicitly models emphintra-layer expert cooperation.
  arXiv  Detail & Related papers  (2026-06-03T08:41:48Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret [71.69884486156359]
  We study episodic RL from emphmulti-source imperfect preferences through a cumulative imperfection budget.<n>We propose a unified algorithm with regret $tildeO(sqrtK/M+)$, which exhibits a best-of-both-regimes behavior.
  arXiv  Detail & Related papers  (2026-03-20T19:34:53Z)
• A Boundary-Metric Evaluation Protocol for Whiteboard Stroke Segmentation Under Extreme Imbalance [0.0]
  Boundary-aware metrics and a thin-subset equity analysis change how loss functions rank and exposes hidden trade-offs.<n>Five losses are trained three times each on a DeepLabV3-MobileNetV3 model and evaluated on 12 held-out images.<n>Overlap-based losses improve F1 by more than 20 points over cross-entropy.
  arXiv  Detail & Related papers  (2026-02-26T05:16:18Z)
• PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective [59.24570811503256]
  We propose PIO-FVLM to reduce redundant visual tokens in vision-models (VLMs) to accelerate inference.<n>The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment.<n>On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance.
  arXiv  Detail & Related papers  (2026-02-04T15:33:10Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs [62.565573316667276]
  We develop an objective that encodes how a sample relates to others. We train vision models based on similarities in class or text caption descriptions. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8%$ on ImageNet and $18.1%$ on ImageNet Real.
  arXiv  Detail & Related papers  (2024-07-25T15:38:16Z)
• Patch-level Routing in Mixture-of-Experts is Provably Sample-efficient for Convolutional Neural Networks [74.68583356645276]
  In deep learning, mixture-of-experts (MoE) activates one or few experts (sub-networks) on a per-sample or per-token basis. We show for the first time that pMoE provably reduces the required number of training samples to achieve desirable generalization.
  arXiv  Detail & Related papers  (2023-06-07T00:16:10Z)
• Expressive Losses for Verified Robustness via Convex Combinations [67.54357965665676]
  We study the relationship between the over-approximation coefficient and performance profiles across different expressive losses. We show that, while expressivity is essential, better approximations of the worst-case loss are not necessarily linked to superior robustness-accuracy trade-offs.
  arXiv  Detail & Related papers  (2023-05-23T12:20:29Z)
• Unlocking Deterministic Robustness Certification on ImageNet [39.439003787779434]
  This paper investigates strategies for expanding certifiably robust training to larger, deeper models. We show that fast ways of bounding the Lipschitz constant for conventional ResNets are loose, and show how to address this by designing a new residual block. We are able to scale up fast deterministic robustness guarantees to ImageNet, demonstrating that this approach to robust learning can be applied to real-world applications.
  arXiv  Detail & Related papers  (2023-01-29T21:40:04Z)
• Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP [52.25478388220691]
  Vision multi-layer perceptrons (MLPs) have shown promising performance in computer vision tasks. They use token-mixing layers to capture cross-token interactions, as opposed to the multi-head self-attention mechanism used by Transformers. We propose a new positional spacial gating unit (PoSGU) to efficiently encode the cross-token relations for token mixing.
  arXiv  Detail & Related papers  (2022-07-15T04:18:06Z)
• Binarizing by Classification: Is soft function really necessary? [4.329951775163721]
  We propose to tackle network binarization as a binary classification problem. We also take binarization as a lightweighting approach for pose estimation models. The proposed method enables binary networks to achieve a mAP of up to $60.6$ for the first time.
  arXiv  Detail & Related papers  (2022-05-16T02:47:41Z)
• Dep-$L_0$: Improving $L_0$-based Network Sparsification via Dependency Modeling [6.081082481356211]
  Training deep neural networks with an $L_0$ regularization is one of the prominent approaches for network pruning or sparsification. We show that this method performs inconsistently on large-scale learning tasks, such as ResNet50 on ImageNet. We propose a dependency modeling of binary gates, which can be modeled effectively as a multi-layer perceptron.
  arXiv  Detail & Related papers  (2021-06-30T19:33:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.