Multi-Domain Learning with Global Expert Mapping

• URL: http://arxiv.org/abs/2604.18842v1
• Date: Mon, 20 Apr 2026 21:09:34 GMT
• Title: Multi-Domain Learning with Global Expert Mapping
• Abstract summary: Mixture-of-Experts (MoE) models provide a scalable solution by routing inputs to specializedworks (experts)<n>We propose GEM (Global Expert Mapping), a planner-compiler framework that replaces the learned router with a global scheduler.<n>Our planner, based on linear programming relaxation, computes a fractional assignment of datasets to experts, while the compiler applies hierarchical rounding to convert this soft plan into a deterministic, capacity-aware mapping.<n>Experiments show that GEM-DINO achieves state-of-the-art performance on the UODB benchmark, with notable gains on underrepresented datasets and solves task interference in few
• Score: 102.62297074508147
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Human perception generalizes well across different domains, but most vision models struggle beyond their training data. This gap motivates multi-dataset learning, where a single model is trained on diverse datasets to improve robustness under domain shifts. However, unified training remains challenging due to inconsistencies in data distributions and label semantics. Mixture-of-Experts (MoE) models provide a scalable solution by routing inputs to specialized subnetworks (experts). Yet, existing MoEs often fail to specialize effectively, as their load-balancing mechanisms enforce uniform input distribution across experts. This fairness conflicts with domain-aware routing, causing experts to learn redundant representations, and reducing performance especially on rare or out-of-distribution domains. We propose GEM (Global Expert Mapping), a planner-compiler framework that replaces the learned router with a global scheduler. Our planner, based on linear programming relaxation, computes a fractional assignment of datasets to experts, while the compiler applies hierarchical rounding to convert this soft plan into a deterministic, capacity-aware mapping. Unlike prior MoEs, GEM avoids balancing loss, resolves the conflict between fairness and specialization, and produces interpretable routing. Experiments show that GEM-DINO achieves state-of-the-art performance on the UODB benchmark, with notable gains on underrepresented datasets and solves task interference in few-shot adaptation scenarios.

Related papers

• HEAT: Heterogeneous End-to-End Autonomous Driving via Trajectory-Guided World Models [50.18051979020699]
  End-to-end autonomous driving has emerged as a compelling alternative to traditional modular pipelines.<n>We propose a trajectory-driven learning paradigm that organizes training around planning trajectories.<n>We show that a single unified model can be trained on heterogeneous datasets while maintaining strong performance within each domain.
  arXiv  Detail & Related papers  (2026-05-19T10:12:01Z)
• Expert Divergence Learning for MoE-based Language Models [21.857507958048696]
  We introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts.<n>Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora.<n>We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch.
  arXiv  Detail & Related papers  (2026-02-10T06:58:38Z)
• OFA-MAS: One-for-All Multi-Agent System Topology Design based on Mixture-of-Experts Graph Generative Models [57.94189874119267]
  Multi-Agent Systems (MAS) offer a powerful paradigm for solving complex problems.<n>Current graph learning-based design methodologies often adhere to a "one-for-one" paradigm.<n>We propose OFA-TAD, a one-for-all framework that generates adaptive collaboration graphs for any task described in natural language.
  arXiv  Detail & Related papers  (2026-01-19T12:23:44Z)
• Adaptive Graph Mixture of Residual Experts: Unsupervised Learning on Diverse Graphs with Heterogeneous Specialization [17.89950704690598]
  We introduce ADaMoRE, a principled framework that enables robust, fully unsupervised training of heterogeneous MoE on graphs.<n>A structurally-aware gating network performs fine-grained node routing.<n>Our design improves data efficiency and training stability.
  arXiv  Detail & Related papers  (2025-10-24T07:18:24Z)
• Let Synthetic Data Shine: Domain Reassembly and Soft-Fusion for Single Domain Generalization [68.41367635546183]
  Single Domain Generalization aims to train models with consistent performance across diverse scenarios using data from a single source.<n>We propose Discriminative Domain Reassembly and Soft-Fusion (DRSF), a training framework leveraging synthetic data to improve model generalization.
  arXiv  Detail & Related papers  (2025-03-17T18:08:03Z)
• Decoupled Training: Return of Frustratingly Easy Multi-Domain Learning [20.17925272562433]
  Multi-domain learning aims to train a model with minimal average risk across multiple overlapping but non-identical domains. We propose Decoupled Training (D-Train) as a frustratingly easy and hyper parameter-free multi-domain learning method. D-Train is a tri-phase general-to-specific training strategy that first pre-trains on all domains to warm up a root model, then post-trains on each domain by splitting into multi-heads, and finally fine-tunes the heads by fixing the backbone.
  arXiv  Detail & Related papers  (2023-09-19T04:06:41Z)
• CHALLENGER: Training with Attribution Maps [63.736435657236505]
  We show that utilizing attribution maps for training neural networks can improve regularization of models and thus increase performance. In particular, we show that our generic domain-independent approach yields state-of-the-art results in vision, natural language processing and on time series tasks.
  arXiv  Detail & Related papers  (2022-05-30T13:34:46Z)
• META: Mimicking Embedding via oThers' Aggregation for Generalizable Person Re-identification [68.39849081353704]
  Domain generalizable (DG) person re-identification (ReID) aims to test across unseen domains without access to the target domain data at training time. This paper presents a new approach called Mimicking Embedding via oThers' Aggregation (META) for DG ReID.
  arXiv  Detail & Related papers  (2021-12-16T08:06:50Z)
• Decentralized Local Stochastic Extra-Gradient for Variational Inequalities [125.62877849447729]
  We consider distributed variational inequalities (VIs) on domains with the problem data that is heterogeneous (non-IID) and distributed across many devices. We make a very general assumption on the computational network that covers the settings of fully decentralized calculations. We theoretically analyze its convergence rate in the strongly-monotone, monotone, and non-monotone settings.
  arXiv  Detail & Related papers  (2021-06-15T17:45:51Z)
• SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalization [7.253255826783766]
  We propose a masking strategy, which determines a continuous weight based on the agreement of gradients that flow in each edge of network. SAND-mask is validated over the Domainbed benchmark for domain generalization.
  arXiv  Detail & Related papers  (2021-06-04T05:20:54Z)
• Generalizable Person Re-identification with Relevance-aware Mixture of Experts [45.13716166680772]
  We propose a novel method called the relevance-aware mixture of experts (RaMoE) RaMoE uses an effective voting-based mixture mechanism to dynamically leverage source domains' diverse characteristics to improve the model's generalization. Considering the target domains' invisibility during training, we propose a novel learning-to-learn algorithm combined with our relation alignment loss to update the voting network.
  arXiv  Detail & Related papers  (2021-05-19T14:19:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.