SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs

• URL: http://arxiv.org/abs/2604.23996v1
• Date: Mon, 27 Apr 2026 03:23:19 GMT
• Title: SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs
• Abstract summary: We propose Soft Modality-guided Expert (SMoES) for large vision models (VLMs)<n>SMoES consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization.<n> Experiments across four MoE-basedVLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency.
• Score: 9.733867547637956
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) has become a prevalent backbone for large vision-language models (VLMs), yet how modality-specific signals should guide expert routing remains under-explored. Existing routing strategies are either hand-crafted or modality-agnostic, relying on idealized priors that ignore the layer-dependent modality fusion patterns in MoE-VLMs and provide little guidance for expert specialization. We propose Soft Modality-guided Expert Specialization (SMoES), which consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization that encourages coherent modality specialization. Our method leverages attention-based or Gaussian-statistics modality scores to optimize mutual information regularization. Experiments across four MoE-based VLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency: 0.9% and 4.2% average gain on multimodal and language-only tasks, 56.1% reduction in EP communication overhead, and 12.3% throughput improvement under realistic deployment. These results validate that aligning routing with modality-aware expert specialization unlocks MoE-VLM capacity and efficiency.

Related papers

• DBES: A Systematic Benchmark and Metric Suite for Evaluating Expert Specialization in Large-Scale MoEs [4.089532981955943]
  Expert specialization in Mixture-of-Experts (MoE) models remains poorly understood.<n>We introduce DBES, a comprehensive diagnostic framework combining a multi-domain benchmark with five theoretically grounded metrics.<n>This work provides the first systematic methodology for evaluating expert specialization independently of accuracy metrics.
  arXiv  Detail & Related papers  (2026-05-18T14:50:30Z)
• Scaling Multi-Node Mixture-of-Experts Inference Using Expert Activation Patterns [10.028141800987548]
  MoE inference at scale is bottlenecked by expert load imbalance and inefficient token routing.<n>We study SOTA open-source MoE models, including Llama 4 Maverick, DeepSeek V3-671B, and Qwen3-230B-A22B.<n>Motivated by these findings, we propose workload-aware micro-batch grouping and an expert placement strategy.
  arXiv  Detail & Related papers  (2026-04-25T05:33:03Z)
• Complementarity-Supervised Spectral-Band Routing for Multimodal Emotion Recognition [60.20529806857076]
  Multimodal emotion recognition fuses cues such as text, video, and audio to understand individual emotional states.<n>Prior methods face two main limitations: mechanically relying on independent unimodal performance, and coarse-grained fusion conflicting with the fine-grained representations required by emotion tasks.<n>We propose the Complementarity-Supervised Multi-Band Expert Network, named Atsuko, to model fine-grained complementary features via multi-scale band decomposition and expert collaboration.
  arXiv  Detail & Related papers  (2026-03-07T03:58:48Z)
• MoDES: Accelerating Mixture-of-Experts Multimodal Large Language Models via Dynamic Expert Skipping [52.02659589971978]
  We propose MoDES, the first training-free framework that adaptively skips experts to enable efficient and accurate MoE MLLM inference.<n>MoDES significantly enhances inference speed, improving the prefilling time by 2.16$times$ and the decoding time by 1.26$times$.
  arXiv  Detail & Related papers  (2025-11-19T18:48:27Z)
• Input Domain Aware MoE: Decoupling Routing Decisions from Task Optimization in Mixture of Experts [19.707274733121412]
  Sparse Mixture of Experts (sMoE) has become a pivotal approach for scaling large vision-language models.<n>We propose Input Domain Aware MoE, a novel routing framework that leverages a probabilistic mixture model to better partition the input space.<n>By modeling routing probabilities as a mixture of distributions, our method enables experts to develop clear specialization boundaries while achieving balanced utilization.
  arXiv  Detail & Related papers  (2025-10-18T11:01:03Z)
• Adaptive Shared Experts with LoRA-Based Mixture of Experts for Multi-Task Learning [49.90176890917986]
  Mixture-of-Experts (MoE) has emerged as a powerful framework for multi-task learning (MTL)<n>Existing MoE-MTL methods often rely on single-task pretrained backbones and suffer from redundant adaptation and inefficient knowledge sharing.<n>We propose adaptive shared experts (ASE) within a low-rank adaptation (LoRA) based MoE, where shared experts are assigned router-computed gating weights jointly normalized with sparse experts.
  arXiv  Detail & Related papers  (2025-10-01T06:49:19Z)
• Joint Learning using Mixture-of-Expert-Based Representation for Enhanced Speech Generation and Robust Emotion Recognition [54.44798086835314]
  Speech emotion recognition (SER) plays a critical role in building emotion-aware speech systems, but its performance degrades significantly under noisy conditions.<n>We propose the Sparse Mixture-of-Experts Representation Integration Technique (Sparse MERIT), a flexible MTL framework that applies frame-wise expert routing over self-supervised speech representations.<n> Experiments on the MSP-Podcast corpus show that Sparse MERIT consistently outperforms baseline models on both SER and SE tasks.
  arXiv  Detail & Related papers  (2025-09-10T10:18:56Z)
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models [52.876185634349575]
  We propose to incorporate Mixture of Intra- and Inter-Modality Experts (MoIIE) to Large Vision-Language Models (LVLMs)<n>For each token, expert routing is guided by its modality, directing tokens to their respective intra-modality experts as well as a shared pool of inter-modality experts.<n>Our MoIIE models with 5.5B and 11.3B activated parameters match or even surpass the performance of existing advanced open-source MoE-LLMs based multi-modal models.
  arXiv  Detail & Related papers  (2025-08-13T13:00:05Z)
• Enhancing CTR Prediction with De-correlated Expert Networks [45.50697497028273]
  We propose a De-Correlated MoE (D-MoE) framework, which introduces a Cross-Expert De-Correlation loss to minimize expert correlations.<n>We show that D-MoE achieves a significant 1.19% Gross Merchandise Volume (GMV) lift compared to the Multi-Embedding MoE baseline.
  arXiv  Detail & Related papers  (2025-05-23T14:04:38Z)
• MoECollab: Democratizing LLM Development Through Collaborative Mixture of Experts [0.76146285961466]
  Large Language Model (LLM) development has become increasingly centralized, limiting participation to well-resourced organizations.<n>This paper introduces MoECollab, a novel framework leveraging Mixture of Experts (MoE) architecture to enable distributed, collaborative LLM development.
  arXiv  Detail & Related papers  (2025-03-16T17:52:40Z)
• Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selection [19.365009652356793]
  Expert-Token Resonance (ETR) is a theoretically-grounded bidirectional routing mechanism that reimagines expert-token interactions.<n>ETR achieves 5.4%-46.6% improvements in end-to-end training efficiency compared to baseline MoE implementations.
  arXiv  Detail & Related papers  (2024-05-24T02:50:44Z)
• T-REX: Mixture-of-Rank-One-Experts with Semantic-aware Intuition for Multi-task Large Language Model Finetuning [31.276142111455847]
  Large language models (LLMs) encounter significant adaptation challenges in diverse multitask finetuning.<n>We design a novel framework, mixunderlinetextbfTureunderlinetextbf-of-underlinetextbfRank-onunderlinetextbfE-eunderlinetextbfXper ts (textttT-REX)<n>Rank-1 experts enable a mix-and-match mechanism to quadratically expand the vector subspace of experts with linear parameter overheads, achieving approximate error reduction with optimal
  arXiv  Detail & Related papers  (2024-04-13T12:14:58Z)
• Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts [74.40198929049959]
  Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. generalist LMMs often suffer from performance degradation when tuned over a large collection of tasks. We propose Omni-SMoLA, an architecture that uses the Soft MoE approach to mix many multimodal low rank experts.
  arXiv  Detail & Related papers  (2023-12-01T23:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.