Do Domain-specific Experts exist in MoE-based LLMs?

• URL: http://arxiv.org/abs/2604.05267v1
• Date: Tue, 07 Apr 2026 00:05:09 GMT
• Title: Do Domain-specific Experts exist in MoE-based LLMs?
• Abstract summary: We propose textbfDomain Steering Mixture of Experts (DSMoE), a training-free framework that introduces zero additional inference cost and outperforms both well-trained MoE-based LLMs and strong baselines.<n>Our method achieves strong performance and robust generalization without increasing inference cost or requiring additional retraining.
• Score: 14.774596844618396
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the era of Large Language Models (LLMs), the Mixture of Experts (MoE) architecture has emerged as an effective approach for training extremely large models with improved computational efficiency. This success builds upon extensive prior research aimed at enhancing expert specialization in MoE-based LLMs. However, the nature of such specializations and how they can be systematically interpreted remain open research challenges. In this work, we investigate this gap by posing a fundamental question: \textit{Do domain-specific experts exist in MoE-based LLMs?} To answer the question, we evaluate ten advanced MoE-based LLMs ranging from 3.8B to 120B parameters and provide empirical evidence for the existence of domain-specific experts. Building on this finding, we propose \textbf{Domain Steering Mixture of Experts (DSMoE)}, a training-free framework that introduces zero additional inference cost and outperforms both well-trained MoE-based LLMs and strong baselines, including Supervised Fine-Tuning (SFT). Experiments on four advanced open-source MoE-based LLMs across both target and non-target domains demonstrate that our method achieves strong performance and robust generalization without increasing inference cost or requiring additional retraining. Our implementation is publicly available at https://github.com/giangdip2410/Domain-specific-Experts.

Related papers

• $Φ$-Bench: Can Large Language Models Engineer the Infrastructure That Powers Them? [38.43131935999474]
  $$-Bench is a benchmark for evaluating large language models (LLMs) on engineering the LLM infrastructure stack.<n>$$-Bench is derived from optimization problems studied in frontier research and grounded in real-world code repositories.
  arXiv  Detail & Related papers  (2026-09-09T14:23:47Z)
• Expert Divergence Learning for MoE-based Language Models [21.857507958048696]
  We introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts.<n>Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora.<n>We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch.
  arXiv  Detail & Related papers  (2026-02-10T06:58:38Z)
• Mixing Expert Knowledge: Bring Human Thoughts Back To the Game of Go [74.28228642327726]
  Large language models (LLMs) have demonstrated exceptional performance in reasoning tasks such as mathematics and coding.<n>LoGos is a powerful LLM that not only maintains outstanding general reasoning abilities, but also conducts Go gameplay in natural language.<n>LoGos achieves performance comparable to human professional players, substantially surpassing all existing LLMs.
  arXiv  Detail & Related papers  (2026-01-23T05:00:49Z)
• General-Reasoner: Advancing LLM Reasoning Across All Domains [64.70599911897595]
  Reinforcement learning (RL) has recently demonstrated strong potential in enhancing the reasoning capabilities of large language models (LLMs)<n>We propose General-Reasoner, a novel training paradigm designed to enhance LLM reasoning capabilities across diverse domains.<n>We train a series of models and evaluate them on a wide range of datasets covering wide domains like physics, chemistry, finance, electronics etc.
  arXiv  Detail & Related papers  (2025-05-20T17:41:33Z)
• Evaluating Expert Contributions in a MoE LLM for Quiz-Based Tasks [0.0]
  We evaluate expert contributions on the quiz-based MMLU benchmark.<n>We show that most experts were never activated during inference on this benchmark.<n>We also demonstrate that the average performance of some experts within the same layer varies significantly.
  arXiv  Detail & Related papers  (2025-02-24T14:23:52Z)
• Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models [24.915387910764082]
  Expert-Specialized Fine-Tuning, or ESFT, tunes the experts most relevant to downstream tasks while freezing the other experts and modules. MoE models with finer-grained experts are more advantageous in selecting the combination of experts that are most relevant to downstream tasks.
  arXiv  Detail & Related papers  (2024-07-02T03:11:13Z)
• Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts [49.950419707905944]
  We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts. Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.
  arXiv  Detail & Related papers  (2024-06-17T19:06:54Z)
• Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models [90.14693869269519]
  MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. This paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques.
  arXiv  Detail & Related papers  (2024-02-22T18:56:07Z)
• PANDA: Preference Adaptation for Enhancing Domain-Specific Abilities of LLMs [49.32067576992511]
  Large language models often fall short of the performance achieved by domain-specific state-of-the-art models. One potential approach to enhance domain-specific capabilities of LLMs involves fine-tuning them using corresponding datasets. We propose Preference Adaptation for Enhancing Domain-specific Abilities of LLMs (PANDA) Our experimental results reveal that PANDA significantly enhances the domain-specific ability of LLMs on text classification and interactive decision tasks.
  arXiv  Detail & Related papers  (2024-02-20T09:02:55Z)
• OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models [44.848642930797155]
  We release OpenMoE, a series of fully open-sourced and reproducible decoder-only Mixture-of-Experts (MoE) based large language models (LLMs) Our investigation confirms that MoE-based LLMs can offer a more favorable cost-effectiveness trade-off than dense LLMs. We find that routing decisions in MoE models are predominantly based on token IDs, with minimal context relevance.
  arXiv  Detail & Related papers  (2024-01-29T12:05:02Z)
• EcomGPT-CT: Continual Pre-training of E-commerce Large Language Models with Semi-structured Data [67.8302955948861]
  Large Language Models (LLMs) pre-trained on massive corpora have exhibited remarkable performance on various NLP tasks. Applying these models to specific domains still poses significant challenges, such as lack of domain knowledge. We focus on domain-specific continual pre-training of LLMs using E-commerce domain as an exemplar.
  arXiv  Detail & Related papers  (2023-12-25T11:31:47Z)
• Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts [74.40198929049959]
  Large multi-modal models (LMMs) exhibit remarkable performance across numerous tasks. generalist LMMs often suffer from performance degradation when tuned over a large collection of tasks. We propose Omni-SMoLA, an architecture that uses the Soft MoE approach to mix many multimodal low rank experts.
  arXiv  Detail & Related papers  (2023-12-01T23:04:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.