HDEE: Heterogeneous Domain Expert Ensemble

• URL: http://arxiv.org/abs/2502.19385v1
• Date: Wed, 26 Feb 2025 18:30:49 GMT
• Title: HDEE: Heterogeneous Domain Expert Ensemble
• Abstract summary: Several studies aim to reduce the dependency on centralization by reducing the communication overhead of training dense models.<n>Taking this idea of reducing communication overhead to a natural extreme, has been shown to outperform large dense models trained in traditional centralized settings.<n>We show that the heterogeneous ensembles achieve the lowest perplexity scores in $20$ out of the $21$ data domains used in the evaluation.
• Score: 0.3641292357963815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training dense LLMs requires enormous amounts of data and centralized compute, which introduces fundamental bottlenecks and ever-growing costs for large models. Several studies aim to reduce this dependency on centralization by reducing the communication overhead of training dense models. Taking this idea of reducing communication overhead to a natural extreme, by training embarrassingly parallelizable ensembles of small independent experts, has been shown to outperform large dense models trained in traditional centralized settings. However, existing studies do not take into account underlying differences amongst data domains and treat them as monolithic, regardless of their underlying complexity, size, or distribution. In this paper, we explore the effects of introducing heterogeneity to these ensembles of domain expert models. Specifically, by allowing models within the ensemble to vary in size--as well as the number of training steps taken depending on the training data's domain--we study the effect heterogeneity has on these ensembles when evaluated against domains included in, and excluded from, the training set. We use the same compute budget to train heterogeneous ensembles and homogeneous baselines for comparison. We show that the heterogeneous ensembles achieve the lowest perplexity scores in $20$ out of the $21$ data domains used in the evaluation. Our code is available at https://github.com/gensyn-ai/hdee.

Related papers

• Explaining Data Mixing Scaling Laws [2.1378946778058796]
  We propose a unified framework to explain the underlying mechanics of data mixing.<n>We identify two key factors that govern the domain losses of models trained on different data mixtures.<n>Our model successfully extrapolates across scales, predicting highly effective mixtures for large, unseen scales.
  arXiv  Detail & Related papers  (2026-06-06T13:31:38Z)
• R&B: Domain Regrouping and Data Mixture Balancing for Efficient Foundation Model Training [11.213419356901005]
  We introduce R&B, a framework that re- Partitions training data based on semantic similarity to create finer-grained domains. Unlike prior works, R&B removes the need for additional compute to obtain evaluation information such as losses or gradients. We demonstrate the effectiveness of R&B on five diverse datasets ranging from natural language to reasoning and multimodal tasks.
  arXiv  Detail & Related papers  (2025-05-01T07:08:19Z)
• Heterogeneity Matters even More in Distributed Learning: Study from Generalization Perspective [14.480713752871523]
  In one-round Federated Learning, $K$ clients have each $n$ training samples generated independently according to a possibly different data distribution.<n>We study the effect of the discrepancy between the clients' data distributions on the generalization error of the aggregated model.<n>It is shown that DSVM generalizes better when the dissimilarity between the clients' training samples is bigger.
  arXiv  Detail & Related papers  (2025-03-03T14:33:38Z)
• Deep Domain Isolation and Sample Clustered Federated Learning for Semantic Segmentation [2.515027627030043]
  In this paper, we explore for the first time the effect of covariate shifts between participants' data in 2D segmentation tasks. We develop Deep Domain Isolation (DDI) to isolate image domains directly in the gradient space of the model. We leverage this clustering algorithm through a Sample Clustered Federated Learning (SCFL) framework.
  arXiv  Detail & Related papers  (2024-10-04T12:43:07Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• FedDisco: Federated Learning with Discrepancy-Aware Collaboration [41.828780724903744]
  We propose a novel aggregation method, Federated Learning with Discrepancy-aware Collaboration (FedDisco) Our FedDisco outperforms several state-of-the-art methods and can be easily incorporated with many existing methods to further enhance the performance.
  arXiv  Detail & Related papers  (2023-05-30T17:20:51Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• Diversity Boosted Learning for Domain Generalization with Large Number of Domains [4.711430413139393]
  We show that Diversity boosted twO-level saMplIng framework helps train robust models against spurious correlations from both domain-side and object-side. We show that DOMI helps train robust models against spurious correlations on rotated MNIST, rotated Fashion MNIST, and iwildcam datasets.
  arXiv  Detail & Related papers  (2022-07-28T02:58:17Z)
• FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data [69.0785021613868]
  Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos. We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies. This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
  arXiv  Detail & Related papers  (2022-05-19T03:32:03Z)
• Differentially-Private Clustering of Easy Instances [67.04951703461657]
  In differentially private clustering, the goal is to identify $k$ cluster centers without disclosing information on individual data points. We provide implementable differentially private clustering algorithms that provide utility when the data is "easy" We propose a framework that allows us to apply non-private clustering algorithms to the easy instances and privately combine the results.
  arXiv  Detail & Related papers  (2021-12-29T08:13:56Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• Examining and Combating Spurious Features under Distribution Shift [94.31956965507085]
  We define and analyze robust and spurious representations using the information-theoretic concept of minimal sufficient statistics. We prove that even when there is only bias of the input distribution, models can still pick up spurious features from their training data. Inspired by our analysis, we demonstrate that group DRO can fail when groups do not directly account for various spurious correlations.
  arXiv  Detail & Related papers  (2021-06-14T05:39:09Z)
• Clustered Federated Learning via Generalized Total Variation Minimization [83.26141667853057]
  We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure. Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV) Our main algorithmic contribution is a fully decentralized federated learning algorithm.
  arXiv  Detail & Related papers  (2021-05-26T18:07:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.