Mixture of Experts in Image Classification: What's the Sweet Spot?

• URL: http://arxiv.org/abs/2411.18322v2
• Date: Fri, 24 Oct 2025 17:36:28 GMT
• Title: Mixture of Experts in Image Classification: What's the Sweet Spot?
• Abstract summary: We study the integration of MoE layers into image classification architectures using open datasets.<n>We find that moderate parameter activation per sample provides the best trade-off between performance and efficiency.<n>MoE layers most effectively strengthen tiny and mid-sized models, while gains taper off for large-capacity networks.
• Score: 41.72573567802606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) models have shown promising potential for parameter-efficient scaling across domains. However, their application to image classification remains limited, often requiring billion-scale datasets to be competitive. In this work, we explore the integration of MoE layers into image classification architectures using open datasets. We conduct a systematic analysis across different MoE configurations and model scales. We find that moderate parameter activation per sample provides the best trade-off between performance and efficiency. However, as the number of activated parameters increases, the benefits of MoE diminish. Our analysis yields several practical insights for vision MoE design. First, MoE layers most effectively strengthen tiny and mid-sized models, while gains taper off for large-capacity networks and do not redefine state-of-the-art ImageNet performance. Second, a Last-2 placement heuristic offers the most robust cross-architecture choice, with Every-2 slightly better for Vision Transform (ViT), and both remaining effective as data and model scale increase. Third, larger datasets (e.g., ImageNet-21k) allow more experts, up to 16, for ConvNeXt to be utilized effectively without changing placement, as increased data reduces overfitting and promotes broader expert specialization. Finally, a simple linear router performs best, suggesting that additional routing complexity yields no consistent benefit.

Related papers

• MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts [41.67455261906954]
  We introduce a scale-aware token-routed Mixture of Experts (MoE) architecture, allowing scale-adaptive expert selection.<n>Unlike naive alignment, we analyse and design a residual feature aggregation scheme tailored to the Visual AutoRegressive modeling paradigm.<n>Our model achieves a superior FID compared to the dense baseline on the ImageNet 256*256 benchmark.
  arXiv  Detail & Related papers  (2026-07-01T03:11:21Z)
• Scaling Text-to-Image Diffusion Transformers with Representation Autoencoders [74.72147962028265]
  Representation Autoencoders (RAEs) have shown distinct advantages in diffusion modeling on ImageNet.<n>We investigate whether this framework can scale to large-scale, freeform text-to-image (T2I) generation.
  arXiv  Detail & Related papers  (2026-01-22T18:58:16Z)
• Beyond ImageNet: Understanding Cross-Dataset Robustness of Lightweight Vision Models [13.660350750023055]
  We present the first systematic evaluation of 11 lightweight vision models (2.5M parameters) trained under a fixed 100-epoch schedule across 7 diverse datasets.<n>We introduce the Cross-Dataset Score (xScore), a unified metric that quantifies the consistency and robustness of model performance across diverse visual domains.
  arXiv  Detail & Related papers  (2025-11-01T00:40:06Z)
• ScaleNet: Scaling up Pretrained Neural Networks with Incremental Parameters [67.87703790962388]
  We introduce ScaleNet, an efficient approach for scaling vision transformers (ViTs)<n>Unlike conventional training from scratch, ScaleNet facilitates rapid model expansion with negligible increases in parameters.<n>We show that ScaleNet achieves a 7.42% accuracy improvement over training from scratch while requiring only one-third of the training epochs.
  arXiv  Detail & Related papers  (2025-10-21T09:07:25Z)
• EGFormer: Towards Efficient and Generalizable Multimodal Semantic Segmentation [6.314084134346798]
  EGFormer is an efficient multimodal semantic segmentation framework.<n>It flexibly integrates an arbitrary number of modalities while significantly reducing model parameters and inference time.<n>It achieves competitive performance with up to 88 percent reduction in parameters and 50 percent fewer GFLOPs.
  arXiv  Detail & Related papers  (2025-05-20T07:08:49Z)
• MoQa: Rethinking MoE Quantization with Multi-stage Data-model Distribution Awareness [12.059149430757863]
  Mix-of-Experts (MoE) has become the main form of Large Language Models (LLMs) MoQa decouples the data-model distribution complexity of MoEs in multiple analysis stages. Experiments show that MoQa achieves a 1.692.18 perplexity decrease in language modeling tasks and a 1.58%8.91% accuracy improvement in zero-shot inference tasks.
  arXiv  Detail & Related papers  (2025-03-27T03:52:25Z)
• Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer [7.230514235208748]
  We propose Union-of-Experts (UoE), which decomposes the transformer model into an equivalent group of experts.<n>In language modeling tasks, UoE achieves an average reduction of 2.38 in perplexity compared to the best-performing MoE method.<n>In image classification, it yields an average accuracy improvement of 1.75% over the best model.
  arXiv  Detail & Related papers  (2025-03-04T11:01:25Z)
• Multi-Scale and Multimodal Species Distribution Modeling [4.022195138381868]
  Species distribution models (SDMs) aim to predict the distribution of species relating occurrence data with environmental variables. Recent applications of deep learning to SDMs have enabled new avenues, specifically the inclusion of spatial data. We develop a modular structure for SDMs that allows us to test the effect of scale in both single- and multi-scale settings. Results on the GeoLifeCLEF 2023 benchmark indicate that considering multimodal data and learning multi-scale representations leads to more accurate models.
  arXiv  Detail & Related papers  (2024-11-06T15:57:20Z)
• Mixture of A Million Experts [1.240096657086732]
  This paper introduces PEER, a novel layer design that utilizes the product key technique for sparse retrieval from a vast pool of experts. Experiments on language modeling tasks demonstrate that PEER layers outperform dense FFWs and coarse-grained MoEs in terms of performance-compute trade-off.
  arXiv  Detail & Related papers  (2024-07-04T20:59:20Z)
• A Provably Effective Method for Pruning Experts in Fine-tuned Sparse Mixture-of-Experts [49.394145046409044]
  This paper provides the first provably efficient technique for pruning experts in finetuned MoE models. We theoretically prove that prioritizing the pruning of the experts with a smaller change of the routers l2 norm from the pretrained model guarantees the preservation of test accuracy. Although our theoretical analysis is centered on binary classification tasks on simplified MoE architecture, our expert pruning method is verified on large vision MoE models.
  arXiv  Detail & Related papers  (2024-05-26T17:52:58Z)
• DEEM: Diffusion Models Serve as the Eyes of Large Language Models for Image Perception [66.88792390480343]
  We propose DEEM, a simple but effective approach that utilizes the generative feedback of diffusion models to align the semantic distributions of the image encoder. DEEM exhibits enhanced robustness and a superior capacity to alleviate model hallucinations while utilizing fewer trainable parameters, less pre-training data, and a smaller base model size.
  arXiv  Detail & Related papers  (2024-05-24T05:46:04Z)
• FissionFusion: Fast Geometric Generation and Hierarchical Souping for Medical Image Analysis [0.7751705157998379]
  The scarcity of well-annotated medical datasets requires leveraging transfer learning from broader datasets like ImageNet or pre-trained models like CLIP. Model soups averages multiple fine-tuned models aiming to improve performance on In-Domain (ID) tasks and enhance robustness against Out-of-Distribution (OOD) datasets. We propose a hierarchical merging approach that involves local and global aggregation of models at various levels.
  arXiv  Detail & Related papers  (2024-03-20T06:48:48Z)
• The Importance of Downstream Networks in Digital Pathology Foundation Models [1.689369173057502]
  We evaluate seven feature extractor models across three different datasets with 162 different aggregation model configurations. We find that the performance of many current feature extractor models is notably similar.
  arXiv  Detail & Related papers  (2023-11-29T16:54:25Z)
• StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data [129.92449761766025]
  We propose a novel data collection methodology that synchronously synthesizes images and dialogues for visual instruction tuning. This approach harnesses the power of generative models, marrying the abilities of ChatGPT and text-to-image generative models. Our research includes comprehensive experiments conducted on various datasets.
  arXiv  Detail & Related papers  (2023-08-20T12:43:52Z)
• Scaling Vision-Language Models with Sparse Mixture of Experts [128.0882767889029]
  We show that mixture-of-experts (MoE) techniques can achieve state-of-the-art performance on a range of benchmarks over dense models of equivalent computational cost. Our research offers valuable insights into stabilizing the training of MoE models, understanding the impact of MoE on model interpretability, and balancing the trade-offs between compute performance when scaling vision-language models.
  arXiv  Detail & Related papers  (2023-03-13T16:00:31Z)
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications [68.35683849098105]
  We introduce split depth-wise transpose attention (SDTA) encoder that splits input tensors into multiple channel groups. Our EdgeNeXt model with 1.3M parameters achieves 71.2% top-1 accuracy on ImageNet-1K. Our EdgeNeXt model with 5.6M parameters achieves 79.4% top-1 accuracy on ImageNet-1K.
  arXiv  Detail & Related papers  (2022-06-21T17:59:56Z)
• Few-Shot Diffusion Models [15.828257653106537]
  We present Few-Shot Diffusion Models (FSDM), a framework for few-shot generation leveraging conditional DDPMs. FSDM is trained to adapt the generative process conditioned on a small set of images from a given class by aggregating image patch information. We empirically show that FSDM can perform few-shot generation and transfer to new datasets.
  arXiv  Detail & Related papers  (2022-05-30T23:20:33Z)
• Oops I Took A Gradient: Scalable Sampling for Discrete Distributions [53.3142984019796]
  We show that this approach outperforms generic samplers in a number of difficult settings. We also demonstrate the use of our improved sampler for training deep energy-based models on high dimensional discrete data.
  arXiv  Detail & Related papers  (2021-02-08T20:08:50Z)
• Robust Finite Mixture Regression for Heterogeneous Targets [70.19798470463378]
  We propose an FMR model that finds sample clusters and jointly models multiple incomplete mixed-type targets simultaneously. We provide non-asymptotic oracle performance bounds for our model under a high-dimensional learning framework. The results show that our model can achieve state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-12T03:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.