MoECollab: Democratizing LLM Development Through Collaborative Mixture of Experts

• URL: http://arxiv.org/abs/2503.12592v1
• Date: Sun, 16 Mar 2025 17:52:40 GMT
• Title: MoECollab: Democratizing LLM Development Through Collaborative Mixture of Experts
• Abstract summary: Large Language Model (LLM) development has become increasingly centralized, limiting participation to well-resourced organizations.<n>This paper introduces MoECollab, a novel framework leveraging Mixture of Experts (MoE) architecture to enable distributed, collaborative LLM development.
• Score: 0.76146285961466
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Model (LLM) development has become increasingly centralized, limiting participation to well-resourced organizations. This paper introduces MoECollab, a novel framework leveraging Mixture of Experts (MoE) architecture to enable distributed, collaborative LLM development. By decomposing monolithic models into specialized expert modules coordinated by a trainable gating network, our framework allows diverse contributors to participate regardless of computational resources. We provide a complete technical implementation with mathematical foundations for expert dynamics, gating mechanisms, and integration strategies. Experiments on multiple datasets demonstrate that our approach achieves accuracy improvements of 3-7% over baseline models while reducing computational requirements by 34%. Expert specialization yields significant domain-specific gains, with improvements from 51% to 88% F1 score in general classification and from 23% to 44% accuracy in news categorization. We formalize the routing entropy optimization problem and demonstrate how proper regularization techniques lead to 14% higher expert utilization rates. These results validate MoECollab as an effective approach for democratizing LLM development through architecturally-supported collaboration.

Related papers

• SLAI T-Rex: Full-Parameter Post-training of the DeepSeek-V4 Family on Ascend SuperPOD [91.71050339479261]
  This report presents an end-to-end optimization practice on the Ascend NPU SuperPOD.<n>We develop a hierarchical optimization framework spanning model-level parallelism, computation-communication orchestration, and low-level kernel execution.<n>The resulting system achieves 34.22% Model FLOPs Utilization (MFU) with a 2.93x improvement over the open-source baseline recipe.
  arXiv  Detail & Related papers  (2026-07-22T13:49:17Z)
• SMoES: Soft Modality-Guided Expert Specialization in MoE-VLMs [9.733867547637956]
  We propose Soft Modality-guided Expert (SMoES) for large vision models (VLMs)<n>SMoES consists of dynamic soft modality scores that capture layer-dependent fusion patterns, an expert binning mechanism aligned with expert-parallel deployment, and an inter-bin mutual information regularization.<n> Experiments across four MoE-basedVLMs and 16 benchmarks demonstrate improvement on both effectiveness and efficiency.
  arXiv  Detail & Related papers  (2026-04-27T03:23:19Z)
• Expert Divergence Learning for MoE-based Language Models [21.857507958048696]
  We introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts.<n>Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora.<n>We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch.
  arXiv  Detail & Related papers  (2026-02-10T06:58:38Z)
• MeltRTL: Multi-Expert LLMs with Inference-time Intervention for RTL Code Generation [0.0]
  MeltRTL is a novel framework that integrates multi-expert attention with inference-time intervention.<n>MeltRTL significantly improves the accuracy of large language models (LLMs) without retraining the base model.<n>We evaluate MeltRTL on the VerilogEval benchmark, achieving 96% synthesizability and 60% functional correctness.
  arXiv  Detail & Related papers  (2026-01-19T12:49:39Z)
• MindGPT-4ov: An Enhanced MLLM via a Multi-Stage Post-Training Paradigm [25.7631608456086]
  MindGPT-4ov is a general post-training paradigm spanning data production, model training, and efficient deployment.<n>It achieves state-of-the-art performance across multiple benchmarks at low cost.<n>MindGPT-4ov also demonstrates superior user experience in vertical domain tasks.
  arXiv  Detail & Related papers  (2025-12-02T16:04:11Z)
• DYNAMIX: RL-based Adaptive Batch Size Optimization in Distributed Machine Learning Systems [2.472349172396126]
  Existing batch size selection approaches rely on static allocation or simplistics that fail to adapt to heterogeneous, dynamic computing environments.<n>We present DYNAmix, a reinforcement learning framework that formulates batch size optimization as a sequen- tial decision-making problem using Proximal Policy Optimiza- tion (PPO)<n>Our approach employs a multi-dimensional state representation encompassing network-level metrics, system-level resource utilization, and training statistical efficiency indicators to enable informed decision-making across diverse computational resources.
  arXiv  Detail & Related papers  (2025-10-09T17:48:24Z)
• MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Models [52.876185634349575]
  We propose to incorporate Mixture of Intra- and Inter-Modality Experts (MoIIE) to Large Vision-Language Models (LVLMs)<n>For each token, expert routing is guided by its modality, directing tokens to their respective intra-modality experts as well as a shared pool of inter-modality experts.<n>Our MoIIE models with 5.5B and 11.3B activated parameters match or even surpass the performance of existing advanced open-source MoE-LLMs based multi-modal models.
  arXiv  Detail & Related papers  (2025-08-13T13:00:05Z)
• Graft: Integrating the Domain Knowledge via Efficient Parameter Synergy for MLLMs [56.76586846269894]
  Multimodal Large Language Models (MLLMs) have achieved success across various domains.<n>Despite its importance, the study of knowledge sharing among domain-specific MLLMs remains largely underexplored.<n>We propose a unified parameter integration framework that enables modular composition of expert capabilities.
  arXiv  Detail & Related papers  (2025-06-30T15:07:41Z)
• Enhancing CTR Prediction with De-correlated Expert Networks [53.05653547330796]
  We propose a De-Correlated MoE (D-MoE) framework, which introduces a Cross-Expert De-Correlation loss to minimize expert correlations.<n>Extensive experiments have been conducted to validate the effectiveness of D-MoE and the de-correlation principle.
  arXiv  Detail & Related papers  (2025-05-23T14:04:38Z)
• FedADP: Unified Model Aggregation for Federated Learning with Heterogeneous Model Architectures [5.348839333572149]
  Traditional Federated Learning (FL) faces significant challenges in terms of efficiency and accuracy, particularly in heterogeneous environments.<n>We propose FedADP, a federated learning framework designed to adapt to client heterogeneity by dynamically adjusting model architectures during aggregation.<n>Our experimental results demonstrate that FedADP significantly outperforms existing methods, such as FlexiFed, achieving an accuracy improvement of up to 23.30%.
  arXiv  Detail & Related papers  (2025-05-10T02:57:07Z)
• Beyond Standard MoE: Mixture of Latent Experts for Resource-Efficient Language Models [10.623996218106564]
  We introduce a novel parameterization methodology that facilitates the mapping of specific experts into a shared latent space. All expert operations are systematically decomposed into two principal components: a shared projection into a lower-dimensional latent space, followed by expert-specific transformations. This factorized approach substantially diminishes parameter count and computational requirements.
  arXiv  Detail & Related papers  (2025-03-29T14:35:34Z)
• Union of Experts: Adapting Hierarchical Routing to Equivalently Decomposed Transformer [7.230514235208748]
  We propose Union-of-Experts (UoE), which decomposes the transformer model into an equivalent group of experts.<n>In language modeling tasks, UoE achieves an average reduction of 2.38 in perplexity compared to the best-performing MoE method.<n>In image classification, it yields an average accuracy improvement of 1.75% over the best model.
  arXiv  Detail & Related papers  (2025-03-04T11:01:25Z)
• Federated Fine-Tuning of LLMs: Framework Comparison and Research Directions [59.5243730853157]
  Federated learning (FL) provides a privacy-preserving solution for fine-tuning pre-trained large language models (LLMs) using distributed private datasets.<n>This article conducts a comparative analysis of three advanced federated LLM (FedLLM) frameworks that integrate knowledge distillation (KD) and split learning (SL) to mitigate these issues.
  arXiv  Detail & Related papers  (2025-01-08T11:37:06Z)
• GMoE: Empowering LLMs Fine-Tuning via MoE Graph Collaboration [39.302800055216764]
  We introduce a novel MoE graph-based framework $textbfGMoE$, aimed at enhancing the collaboration among multiple experts.<n>In GMoE, a graph router function is designed to capture the collaboration signals among experts.<n>We put forward two coordination strategies in GMoE: the $textitPoisson distribution-based distinction strategy and the $textitNormal distribution-based balance strategy.
  arXiv  Detail & Related papers  (2024-12-18T02:18:57Z)
• MALT: Improving Reasoning with Multi-Agent LLM Training [66.9481561915524]
  MALT (Multi-Agent LLM Training) is a novel post-training strategy that divides the reasoning process into generation, verification, and refinement steps.<n>On MATH, GSM8K, and CSQA, MALT surpasses the same baseline LLM with a relative improvement of 15.66%, 7.42%, and 9.40% respectively.
  arXiv  Detail & Related papers  (2024-12-02T19:30:36Z)
• Combining Domain and Alignment Vectors to Achieve Better Knowledge-Safety Trade-offs in LLMs [64.83462841029089]
  We introduce an efficient merging-based alignment method called textscMergeAlign that interpolates the domain and alignment vectors, creating safer domain-specific models. We apply textscMergeAlign on Llama3 variants that are experts in medicine and finance, obtaining substantial alignment improvements with minimal to no degradation on domain-specific benchmarks.
  arXiv  Detail & Related papers  (2024-11-11T09:32:20Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models [50.331708897857574]
  We introduce FactorLLM, a novel approach that decomposes well-trained dense FFNs into sparse sub-networks without requiring any further modifications. FactorLLM achieves comparable performance to the source model securing up to 85% model performance while obtaining over a 30% increase in inference speed.
  arXiv  Detail & Related papers  (2024-08-15T16:45:16Z)
• Expert-Token Resonance MoE: Bidirectional Routing with Efficiency Affinity-Driven Active Selection [16.062265609569003]
  Mixture-of-Experts (MoE) architectures have emerged as a paradigm-shifting approach for large language models (LLMs)<n>We propose a novel expert routing framework that incorporates: (1) An efficient routing mechanism with lightweight computation; (2) An adaptive bidirectional selection mechanism leveraging resonance between experts and tokens; and (3) A module that determines the lower bounds of expert capacity based on dynamic token distribution analysis.
  arXiv  Detail & Related papers  (2024-05-24T02:50:44Z)
• Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
  We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
  arXiv  Detail & Related papers  (2024-05-18T12:16:01Z)
• Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts [4.629608387540524]
  We present a novel shortcut-connected MoE (ScMoE) architecture with an overlapping parallel strategy. ScMoE allows for a substantial overlap of 70% to 100% with computation. Building on the ScMoE architecture, we further implement an expert offloading strategy to facilitate memory-limited inference.
  arXiv  Detail & Related papers  (2024-04-07T17:17:23Z)
• FedDM: Iterative Distribution Matching for Communication-Efficient Federated Learning [87.08902493524556]
  Federated learning(FL) has recently attracted increasing attention from academia and industry. We propose FedDM to build the global training objective from multiple local surrogate functions. In detail, we construct synthetic sets of data on each client to locally match the loss landscape from original data.
  arXiv  Detail & Related papers  (2022-07-20T04:55:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.