Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling

• URL: http://arxiv.org/abs/2604.13508v2
• Date: Fri, 17 Apr 2026 03:05:14 GMT
• Title: Enhancing Mixture-of-Experts Specialization via Cluster-Aware Upcycling
• Abstract summary: Cluster-aware Upcycling breaks expert symmetry and encourages early specialization aligned with the data distribution.<n>When evaluated on CLIP ViT-B/32 and ViT-B/16, Cluster-aware Upcycling consistently outperforms existing methods.
• Score: 64.1662577506775
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Sparse Upcycling provides an efficient way to initialize a Mixture-of-Experts (MoE) model from pretrained dense weights instead of training from scratch. However, since all experts start from identical weights and the router is randomly initialized, the model suffers from expert symmetry and limited early specialization. We propose Cluster-aware Upcycling, a strategy that incorporates semantic structure into MoE initialization. Our method first partitions the dense model's input activations into semantic clusters. Each expert is then initialized using the subspace representations of its corresponding cluster via truncated SVD, while setting the router's initial weights to the cluster centroids. This cluster-aware initialization breaks expert symmetry and encourages early specialization aligned with the data distribution. Furthermore, we introduce an expert-ensemble self-distillation loss that stabilizes training by providing reliable routing guidance using an ensemble teacher. When evaluated on CLIP ViT-B/32 and ViT-B/16, Cluster-aware Upcycling consistently outperforms existing methods across both zero-shot and few-shot benchmarks. The proposed method also produces more diverse and disentangled expert representations, reduces inter-expert similarity, and leads to more confident routing behavior. Project page: https://sanghyeokchu.github.io/cluster-aware-upcycling/

Related papers

• Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts [27.40662720929157]
  Mixture-of-Experts (MoE) has become the dominant architecture for scaling large language models.<n>We propose expert upcycling, a method for progressively expanding MoE capacity by increasing the number of experts.<n>In our 7B-13B total parameter experiments, the upcycled model matches the fixed-size baseline on validation loss while saving 32% of GPU hours.
  arXiv  Detail & Related papers  (2026-04-21T05:53:33Z)
• DiEC: Diffusion Embedded Clustering [0.76629754443761]
  Deep clustering critically depends on representations that expose clear cluster structure.<n>Most prior methods learn a single embedding with an autoencoder or a self-supervised encoder and treat it as the primary representation for clustering.<n>We propose Embedded Diffusion Clustering (DiEC), an unsupervised clustering framework that exploits this trajectory by directly leveraging intermediate activations of a pretrained diffusion U-Net.
  arXiv  Detail & Related papers  (2025-12-24T03:10:00Z)
• Breaking the MoE LLM Trilemma: Dynamic Expert Clustering with Structured Compression [14.086434595924716]
  Mixture-of-Experts (MoE) Large Language Models (LLMs) face a trilemma of load imbalance, parameter redundancy, and communication overhead.<n>We introduce a unified framework based on dynamic expert clustering and structured compression to address these issues cohesively.
  arXiv  Detail & Related papers  (2025-09-27T10:45:58Z)
• Self-Enhanced Image Clustering with Cross-Modal Semantic Consistency [57.961869351897384]
  We propose a framework based on cross-modal semantic consistency for efficient image clustering.<n>Our framework first builds a strong foundation via Cross-Modal Semantic Consistency.<n>In the first stage, we train lightweight clustering heads to align with the rich semantics of the pre-trained model.<n>In the second stage, we introduce a Self-Enhanced fine-tuning strategy.
  arXiv  Detail & Related papers  (2025-08-02T08:12:57Z)
• Towards Learnable Anchor for Deep Multi-View Clustering [49.767879678193005]
  In this paper, we propose the Deep Multi-view Anchor Clustering (DMAC) model that performs clustering in linear time.<n>With the optimal anchors, the full sample graph is calculated to derive a discriminative embedding for clustering.<n>Experiments on several datasets demonstrate superior performance and efficiency of DMAC compared to state-of-the-art competitors.
  arXiv  Detail & Related papers  (2025-03-16T09:38:11Z)
• Deep Clustering using Dirichlet Process Gaussian Mixture and Alpha Jensen-Shannon Divergence Clustering Loss [0.65268245109828]
  In the autoencoder based deep clustering, the challenge is how to jointly optimize both clustering and dimension reduction together.<n>We introduce an infinite cluster representation using Dirichlet process Gaussian mixture model for joint clustering and model selection in the latent space.<n>We evaluate our proposed deep model selection method with traditional model selection on large class number datasets such as MIT67 and CIFAR100.
  arXiv  Detail & Related papers  (2024-12-12T05:02:41Z)
• Sparser, Better, Deeper, Stronger: Improving Sparse Training with Exact Orthogonal Initialization [49.06421851486415]
  Static sparse training aims to train sparse models from scratch, achieving remarkable results in recent years. We propose Exact Orthogonal Initialization (EOI), a novel sparse Orthogonal Initialization scheme based on random Givens rotations. Our method enables training highly sparse 1000-layer and CNN networks without residual connections or normalization techniques.
  arXiv  Detail & Related papers  (2024-06-03T19:44:47Z)
• Task-customized Masked AutoEncoder via Mixture of Cluster-conditional Experts [104.9871176044644]
  Masked Autoencoder(MAE) is a prevailing self-supervised learning method that achieves promising results in model pre-training. We propose a novel MAE-based pre-training paradigm, Mixture of Cluster-conditional Experts (MoCE) MoCE trains each expert only with semantically relevant images by using cluster-conditional gates.
  arXiv  Detail & Related papers  (2024-02-08T03:46:32Z)
• End-to-end Learnable Clustering for Intent Learning in Recommendation [54.157784572994316]
  We propose a novel intent learning method termed underlineELCRec. It unifies behavior representation learning into an underlineEnd-to-end underlineLearnable underlineClustering framework. We deploy this method on the industrial recommendation system with 130 million page views and achieve promising results.
  arXiv  Detail & Related papers  (2024-01-11T15:22:55Z)
• An Improved Algorithm for Clustered Federated Learning [29.166363192740768]
  This paper addresses the dichotomy between heterogeneous models and simultaneous training in Federated Learning (FL) We define a new clustering model for FL based on the (optimal) local models of the users. textttSR-FCA uses a robust learning algorithm within each cluster to exploit simultaneous training and to correct clustering errors.
  arXiv  Detail & Related papers  (2022-10-20T19:14:36Z)
• Online Deep Clustering for Unsupervised Representation Learning [108.33534231219464]
  Online Deep Clustering (ODC) performs clustering and network update simultaneously rather than alternatingly. We design and maintain two dynamic memory modules, i.e., samples memory to store samples labels and features, and centroids memory for centroids evolution. In this way, labels and the network evolve shoulder-to-shoulder rather than alternatingly.
  arXiv  Detail & Related papers  (2020-06-18T16:15:46Z)
• An Efficient Framework for Clustered Federated Learning [26.24231986590374]
  We address the problem of federated learning (FL) where users are distributed into clusters. We propose the Iterative Federated Clustering Algorithm (IFCA) We show that our algorithm is efficient in non- partitioned problems such as neural networks.
  arXiv  Detail & Related papers  (2020-06-07T08:48:59Z)
• Open-Set Recognition with Gaussian Mixture Variational Autoencoders [91.3247063132127]
  In inference, open-set classification is to either classify a sample into a known class from training or reject it as an unknown class. We train our model to cooperatively learn reconstruction and perform class-based clustering in the latent space. Our model achieves more accurate and robust open-set classification results, with an average F1 improvement of 29.5%.
  arXiv  Detail & Related papers  (2020-06-03T01:15:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.