Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts

• URL: http://arxiv.org/abs/2506.12597v1
• Date: Sat, 14 Jun 2025 18:34:38 GMT
• Title: Automatic Expert Discovery in LLM Upcycling via Sparse Interpolated Mixture-of-Experts
• Abstract summary: SIMoE is an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model.<n>During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint.
• Score: 6.091286069993439
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Sparse Interpolated Mixture-of-Experts (SIMoE) instruction-tuning, an end-to-end algorithm designed to fine-tune a dense pre-trained Large Language Model (LLM) into a MoE-style model that possesses capabilities in multiple specialized domains. During instruction-tuning, SIMoE automatically identifies multiple specialized experts under a specified sparsity constraint, with each expert representing a structurally sparse subset of the seed LLM's parameters that correspond to domain-specific knowledge within the data. SIMoE simultaneously learns an input-dependent expert merging strategy via a router network, leveraging rich cross-expert knowledge for superior downstream generalization that surpasses existing baselines. Empirically, SIMoE consistently achieves state-of-the-art performance on common instruction-tuning benchmarks while maintaining an optimal performance-compute trade-off compared to all baselines.

Related papers

• Structure Aware Neural Architecture Search for Mixture of Experts [3.100317701802537]
  We propose an architecture search framework that makes the alignment between experts and the structure of the data emerge as a search variable.<n>We show that it coincides with the incomplete-data maximum likelihood of a latent-variable mixture, and solve it by a generalised Expectation-Maximisation procedure.<n>We prove that the iterates converge whenever the surrogate errors are summable, and that at every limit point no candidate the search produces improves the true objective.
  arXiv  Detail & Related papers  (2026-08-30T14:37:23Z)
• Multi-level context Modeling for consistent expert selection in Mixture-of-Experts [3.4913929232937604]
  Mixture-of-Experts (MoE) enables efficient scaling of Transformer models by routing tokens to a small subset of experts.<n>Existing routers typically condition expert selection on shallow or isolated token representations, which often produce unstable and semantically inconsistent routing decisions across layers.<n>We propose Multi-level Context Fusion MOE, a framework that constructs context-aware representations by integrating complementary signals from cross-layer semantic aggregation and local token-level interactions.
  arXiv  Detail & Related papers  (2026-07-17T18:22:42Z)
• OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts Graph Generative Models [57.94189874119267]
  Multi-Agent Systems (MAS) offer a powerful paradigm for solving complex problems.<n>Current graph learning-based design methodologies often adhere to a "one-for-one" paradigm.<n>We propose OFA-TAD, a one-for-all framework that generates adaptive collaboration graphs for any task described in natural language.
  arXiv  Detail & Related papers  (2026-01-19T12:23:44Z)
• MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts [12.42628977620548]
  MoST (Mixture of Speech and Text) is a novel large language model that seamlessly integrates speech and text processing.<n>We introduce specialized routing pathways that direct tokens to modality-appropriate experts based on input type.<n>MoST consistently outperforms existing models of comparable parameter counts.
  arXiv  Detail & Related papers  (2026-01-15T10:43:29Z)
• How Many Experts Are Enough? Towards Optimal Semantic Specialization for Mixture-of-Experts [30.125087273625123]
  We propose a semanticaware MoE framework for adaptive expert expansion and dynamic routing.<n>MASS converges to the point of optimal balance between cost-performance trade-off and notably improved sematic specialization.
  arXiv  Detail & Related papers  (2025-12-21T05:37:42Z)
• OrdMoE: Preference Alignment via Hierarchical Expert Group Ranking in Multimodal Mixture-of-Experts LLMs [22.92427011496289]
  We propose OrdMoE, a novel preference alignment framework that bypasses the reliance on external human preferences.<n>OrdMoE constructs an internal preference hierarchy by grouping experts into ranked tiers based on their per-token routing scores.<n>This yields a zero-cost, self-supervised preference ordering over generated responses.
  arXiv  Detail & Related papers  (2025-11-24T11:59:31Z)
• RouteMark: A Fingerprint for Intellectual Property Attribution in Routing-based Model Merging [69.2230254959204]
  We propose RouteMark, a framework for IP protection in merged MoE models.<n>Our key insight is that task-specific experts exhibit stable and distinctive routing behaviors under probing inputs.<n>For attribution and tampering detection, we introduce a similarity-based matching algorithm.
  arXiv  Detail & Related papers  (2025-08-03T14:51:58Z)
• Token-Level Prompt Mixture with Parameter-Free Routing for Federated Domain Generalization [51.562474873972086]
  Federated domain generalization (FedDG) aims to learn a globally generalizable model from decentralized clients with heterogeneous data.<n>Recent studies have introduced prompt learning to adapt vision-language models (VLMs) in FedDG by learning a single global prompt.<n>We propose TRIP, a Token-level prompt mixture with parameter-free routing framework for FedDG.
  arXiv  Detail & Related papers  (2025-04-29T11:06:03Z)
• Symbolic Mixture-of-Experts: Adaptive Skill-based Routing for Heterogeneous Reasoning [76.10639521319382]
  We propose Symbolic-MoE, a symbolic, text-based, and gradient-free Mixture-of-Experts framework.<n>We show that Symbolic-MoE's instance-level expert selection improves performance by a large margin but -- when implemented naively -- can introduce a high computational overhead.
  arXiv  Detail & Related papers  (2025-03-07T18:03:13Z)
• Convergence Rates for Softmax Gating Mixture of Experts [78.3687645289918]
  Mixture of experts (MoE) has emerged as an effective framework to advance the efficiency and scalability of machine learning models.<n>Central to the success of MoE is an adaptive softmax gating mechanism which takes responsibility for determining the relevance of each expert to a given input and then dynamically assigning experts their respective weights.<n>We perform a convergence analysis of parameter estimation and expert estimation under the MoE equipped with the standard softmax gating or its variants, including a dense-to-sparse gating and a hierarchical softmax gating.
  arXiv  Detail & Related papers  (2025-03-05T06:11:24Z)
• Personalized Federated Fine-Tuning for LLMs via Data-Driven Heterogeneous Model Architectures [15.645254436094055]
  Federated Learning (FL) enables collaborative fine-tuning of Large Language Models without accessing raw data.<n>We propose FedAMoLE, a lightweight personalized FL framework that enables data-driven heterogeneous model architectures.<n> Experiments show that FedAMoLE improves client-side performance by an average of 5.14% compared to existing approaches.
  arXiv  Detail & Related papers  (2024-11-28T13:20:38Z)
• Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging [36.0133566024214]
  Upcycling Instruction Tuning (UpIT) is a data-efficient approach for tuning a dense pre-trained model into a MoE instruction model. To ensure each specialized expert in the MoE model works as expected, we select a small amount of seed data that each expert excels to pre-optimize the router.
  arXiv  Detail & Related papers  (2024-10-02T14:48:22Z)
• Self-MoE: Towards Compositional Large Language Models with Self-Specialized Experts [49.950419707905944]
  We present Self-MoE, an approach that transforms a monolithic LLM into a compositional, modular system of self-specialized experts. Our approach leverages self-specialization, which constructs expert modules using self-generated synthetic data. Our findings highlight the critical role of modularity, the applicability of Self-MoE to multiple base LLMs, and the potential of self-improvement in achieving efficient, scalable, and adaptable systems.
  arXiv  Detail & Related papers  (2024-06-17T19:06:54Z)
• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
  We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
  arXiv  Detail & Related papers  (2024-05-18T12:16:01Z)
• Multi-Task Dense Prediction via Mixture of Low-Rank Experts [35.11968315125389]
  We present a novel decoder-focused method for multi-task dense prediction, called Mixture-of-Low-Rank-Experts (MLoRE) To model the global task relationships, MLoRE adds a generic convolution path to the original MoE structure, where each task feature can go through this path for explicit parameter sharing. Our experiments show that our MLoRE achieves superior performance compared to previous state-of-the-art methods on all metrics.
  arXiv  Detail & Related papers  (2024-03-26T14:40:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.