MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

• URL: http://arxiv.org/abs/2605.05225v1
• Date: Sun, 19 Apr 2026 07:25:39 GMT
• Title: MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference
• Abstract summary: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect.<n>We propose MACS (Modality-Aware Capacity Scaling), a training-free inference framework.<n>Specifically, MACS introduces an Entropy-Weighted Load mechanism to quantify the semantic value of visual tokens, addressing information Heterogeneity. Additionally, the Dynamic Modality-Adaptive Capacity mechanism allocates expert resources based on the real-time modal composition of the input.
• Score: 25.81851757648691
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect. This issue is worsened in the multimodal context, as existing token-count-based load balancing methods fail to address two unique challenges: (1) Information Heterogeneity, where numerous redundant visual tokens are treated equally to semantically critical ones, and (2) Modality Dynamics, where varying visual to text ratios across tasks lead to resource misallocation. To address these challenges, we propose MACS (Modality-Aware Capacity Scaling), a training-free inference framework. Specifically, MACS introduces an Entropy-Weighted Load mechanism to quantify the semantic value of visual tokens, addressing information heterogeneity. Additionally, the Dynamic Modality-Adaptive Capacity mechanism allocates expert resources based on the real-time modal composition of the input. Extensive experiments demonstrate that MACS significantly outperforms existing methods on various multimodal benchmarks, providing a novel and robust solution for the efficient deployment of MoE MLLMs in EP inference.

Related papers

• MAF: Multimodal Adaptive Few-shot Prompting for Sentiment Analysis with MLLMs [0.0]
  We propose a Multimodal Adaptive Few-Shot Prompting (MAF) framework for sentiment analysis.<n>MAF retrieves and integrates query-relevant demonstrations to elicit the sentiment reasoning capabilities of MLLMs in a context-sensitive manner.<n>Experiments on public benchmark datasets demonstrate that MAF achieves substantial and consistent performance improvements.
  arXiv  Detail & Related papers  (2026-06-14T09:16:36Z)
• AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba [56.52470564147458]
  We propose textbfAlignMamba-2, an effective and efficient framework for multimodal fusion and sentiment analysis.<n>Our approach introduces a dual alignment strategy that regularizes the model using both Optimal Transport distance and Maximum Mean Discrepancy.<n>More importantly, we design a Modality-Aware Mamba layer, which employs a Mixture-of-Experts architecture with modality-specific and modality-shared experts.
  arXiv  Detail & Related papers  (2026-03-19T03:47:21Z)
• MASQuant: Modality-Aware Smoothing Quantization for Multimodal Large Language Models [16.69997403621672]
  Modality-Aware Smoothing Quantization (MASQuant) is a novel framework that introduces Modality-Aware Smoothing (MAS)<n>MASQuant demonstrates stable quantization performance across both dual-modal and tri-modal MLLMs.<n> Experimental results show that MASQuant is competitive among the state-of-the-art PTQ algorithms.
  arXiv  Detail & Related papers  (2026-03-05T04:41:32Z)
• ELMM: Efficient Lightweight Multimodal Large Language Models for Multimodal Knowledge Graph Completion [34.49091265125411]
  Multimodal Knowledge Graphs (MKGs) extend traditional knowledge graphs by incorporating visual and textual modalities, enabling richer and more expressive entity representations.<n>Existing MKGs often suffer from incompleteness, which hinder their effectiveness in downstream tasks.<n>Large language models (LLMs) have shown promise for knowledge graph completion (KGC), their application to the multimodal setting remains underexplored.<n>We propose Efficient Lightweight Multimodal Language Models (ELMM) for MKGC.
  arXiv  Detail & Related papers  (2025-10-19T08:29:43Z)
• Explaining multimodal LLMs via intra-modal token interactions [55.27436637894534]
  Multimodal Large Language Models (MLLMs) have achieved remarkable success across diverse vision-language tasks, yet their internal decision-making mechanisms remain insufficiently understood.<n>We propose enhancing interpretability by leveraging intra-modal interaction.
  arXiv  Detail & Related papers  (2025-09-26T14:39:13Z)
• FindRec: Stein-Guided Entropic Flow for Multi-Modal Sequential Recommendation [57.577843653775]
  We propose textbfFindRec (textbfFlexible unified textbfinformation textbfdisentanglement for multi-modal sequential textbfRecommendation)<n>A Stein kernel-based Integrated Information Coordination Module (IICM) theoretically guarantees distribution consistency between multimodal features and ID streams.<n>A cross-modal expert routing mechanism that adaptively filters and combines multimodal features based on their contextual relevance.
  arXiv  Detail & Related papers  (2025-07-07T04:09:45Z)
• MoCa: Modality-aware Continual Pre-training Makes Better Bidirectional Multimodal Embeddings [75.0617088717528]
  MoCa is a framework for transforming pre-trained VLM backbones into effective bidirectional embedding models.<n>MoCa consistently improves performance across MMEB and ViDoRe-v2 benchmarks, achieving new state-of-the-art results.
  arXiv  Detail & Related papers  (2025-06-29T06:41:00Z)
• DiffMoE: Dynamic Token Selection for Scalable Diffusion Transformers [86.5541501589166]
  DiffMoE is a batch-level global token pool that enables experts to access global token distributions during training.<n>It achieves state-of-the-art performance among diffusion models on ImageNet benchmark.<n>The effectiveness of our approach extends beyond class-conditional generation to more challenging tasks such as text-to-image generation.
  arXiv  Detail & Related papers  (2025-03-18T17:57:07Z)
• Robust Multi-View Learning via Representation Fusion of Sample-Level Attention and Alignment of Simulated Perturbation [61.64052577026623]
  Real-world multi-view datasets are often heterogeneous and imperfect.<n>We propose a novel robust MVL method (namely RML) with simultaneous representation fusion and alignment.<n>Our RML is self-supervised and can also be applied for downstream tasks as a regularization.
  arXiv  Detail & Related papers  (2025-03-06T07:01:08Z)
• GSIFN: A Graph-Structured and Interlaced-Masked Multimodal Transformer-based Fusion Network for Multimodal Sentiment Analysis [0.0]
  Multimodal Sentiment Analysis (MSA) leverages multiple data modals to analyze human sentiment.<n>Existing MSA models generally employ cutting-edge multimodal fusion and representation learning-based methods to promote MSA capability.<n>Our proposed GSIFN incorporates two main components to solve these problems: (i) a graph-structured and interlaced-masked multimodal Transformer.<n>It adopts the Interlaced Mask mechanism to construct robust multimodal graph embedding, achieve all-modal-in-one Transformer-based fusion, and greatly reduce the computational overhead.
  arXiv  Detail & Related papers  (2024-08-27T06:44:28Z)
• Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities [76.08541852988536]
  We propose to use invariant features for a missing modality imagination network (IF-MMIN) We show that the proposed model outperforms all baselines and invariantly improves the overall emotion recognition performance under uncertain missing-modality conditions.
  arXiv  Detail & Related papers  (2022-10-27T12:16:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.