M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework
- URL: http://arxiv.org/abs/2404.18465v3
- Date: Sun, 12 May 2024 13:11:29 GMT
- Title: M3oE: Multi-Domain Multi-Task Mixture-of Experts Recommendation Framework
- Authors: Zijian Zhang, Shuchang Liu, Jiaao Yu, Qingpeng Cai, Xiangyu Zhao, Chunxu Zhang, Ziru Liu, Qidong Liu, Hongwei Zhao, Lantao Hu, Peng Jiang, Kun Gai,
- Abstract summary: M3oE is an adaptive Multi-domain Multi-task Mixture-of-Experts recommendation framework.
We leverage three mixture-of-experts modules to learn common, domain-aspect, and task-aspect user preferences.
We design a two-level fusion mechanism for precise control over feature extraction and fusion across diverse domains and tasks.
- Score: 32.68911775382326
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Multi-domain recommendation and multi-task recommendation have demonstrated their effectiveness in leveraging common information from different domains and objectives for comprehensive user modeling. Nonetheless, the practical recommendation usually faces multiple domains and tasks simultaneously, which cannot be well-addressed by current methods. To this end, we introduce M3oE, an adaptive Multi-domain Multi-task Mixture-of-Experts recommendation framework. M3oE integrates multi-domain information, maps knowledge across domains and tasks, and optimizes multiple objectives. We leverage three mixture-of-experts modules to learn common, domain-aspect, and task-aspect user preferences respectively to address the complex dependencies among multiple domains and tasks in a disentangled manner. Additionally, we design a two-level fusion mechanism for precise control over feature extraction and fusion across diverse domains and tasks. The framework's adaptability is further enhanced by applying AutoML technique, which allows dynamic structure optimization. To the best of the authors' knowledge, our M3oE is the first effort to solve multi-domain multi-task recommendation self-adaptively. Extensive experiments on two benchmark datasets against diverse baselines demonstrate M3oE's superior performance. The implementation code is available to ensure reproducibility.
Related papers
- Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation [59.41178047749177]
We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training.
We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling.
We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE.
arXiv Detail & Related papers (2024-07-01T09:45:22Z) - Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts [54.529880848937104]
We develop a unified MLLM with the MoE architecture, named Uni-MoE, that can handle a wide array of modalities.
Specifically, it features modality-specific encoders with connectors for a unified multimodal representation.
We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets.
arXiv Detail & Related papers (2024-05-18T12:16:01Z) - Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning [50.73666458313015]
Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications.
MoE has been emerged as a promising solution with its sparse architecture for effective task decoupling.
Intuition-MoR1E achieves superior efficiency and 2.15% overall accuracy improvement across 14 public datasets.
arXiv Detail & Related papers (2024-04-13T12:14:58Z) - Large-Scale Multi-Domain Recommendation: an Automatic Domain Feature Extraction and Personalized Integration Framework [30.46152832695426]
We propose an Automatic Domain Feature Extraction and Personalized Integration (DFEI) framework for the large-scale multi-domain recommendation.
The framework automatically transforms the behavior of each individual user into an aggregation of all user behaviors within the domain, which serves as the domain features.
Experimental results on both public and industrial datasets, consisting of over 20 domains, clearly demonstrate that the proposed framework achieves significantly better performance compared with SOTA baselines.
arXiv Detail & Related papers (2024-04-12T09:57:17Z) - An Aligning and Training Framework for Multimodal Recommendations [23.952221685501875]
multimodal recommendations can leverage rich contexts beyond user and item interactions.
Existing methods mainly use them to help learn ID features.
There exist semantic gaps among multimodal content features and ID features.
arXiv Detail & Related papers (2024-03-19T02:49:32Z) - Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language
Models [7.966452497550907]
We propose the Mixture-of-LoRAs (MoA) architecture for multi-task learning with large language models (LLMs)
Multiple domain-specific LoRA modules can be aligned with the expert design principles observed in Mixture-of-Experts (MoE)
Each LoRA model can be iteratively adapted to a new domain, allowing for quick domain-specific adaptation.
arXiv Detail & Related papers (2024-03-06T03:33:48Z) - Multimodal Instruction Tuning with Conditional Mixture of LoRA [54.65520214291653]
This paper introduces a novel approach that integrates multimodal instruction tuning with Low-Rank Adaption (LoRA)
It innovates upon LoRA by dynamically constructing low-rank adaptation matrices tailored to the unique demands of each input instance.
Experimental results on various multimodal evaluation datasets indicate that MixLoRA not only outperforms the conventional LoRA with the same or even higher ranks.
arXiv Detail & Related papers (2024-02-24T20:15:31Z) - An Unified Search and Recommendation Foundation Model for Cold-Start
Scenario [15.192845741415738]
In commercial search engines and recommendation systems, data from multiple domains is available to jointly train the multi-domain model.
We propose a novel framework called S&R Multi-Domain Foundation, which uses LLM to extract domain invariant features.
We apply the S&R Multi-Domain foundation model to cold start scenarios in the pretrain-finetune manner, which achieves better performance than other SOTA transfer learning methods.
arXiv Detail & Related papers (2023-09-16T10:00:02Z) - Automatic Expert Selection for Multi-Scenario and Multi-Task Search [41.47107282896807]
We propose a novel Automatic Expert Selection framework for Multi-scenario and Multi-task search, named AESM2.
Experiments over two real-world large-scale datasets demonstrate the effectiveness of AESM2 over a battery of strong baselines.
Online A/B test also shows substantial performance gain on multiple metrics.
arXiv Detail & Related papers (2022-05-28T03:41:25Z) - Dual Metric Learning for Effective and Efficient Cross-Domain
Recommendations [85.6250759280292]
Cross domain recommender systems have been increasingly valuable for helping consumers identify useful items in different applications.
Existing cross-domain models typically require large number of overlap users, which can be difficult to obtain in some applications.
We propose a novel cross-domain recommendation model based on dual learning that transfers information between two related domains in an iterative manner.
arXiv Detail & Related papers (2021-04-17T09:18:59Z) - Multi-path Neural Networks for On-device Multi-domain Visual
Classification [55.281139434736254]
This paper proposes a novel approach to automatically learn a multi-path network for multi-domain visual classification on mobile devices.
The proposed multi-path network is learned from neural architecture search by applying one reinforcement learning controller for each domain to select the best path in the super-network created from a MobileNetV3-like search space.
The determined multi-path model selectively shares parameters across domains in shared nodes while keeping domain-specific parameters within non-shared nodes in individual domain paths.
arXiv Detail & Related papers (2020-10-10T05:13:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.