MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation

• URL: http://arxiv.org/abs/2510.15286v1
• Date: Fri, 17 Oct 2025 03:50:09 GMT
• Title: MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation
• Abstract summary: We propose textbfMTmixAtt, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention.<n>The textbfAutoToken module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups.<n>Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines.
• Score: 8.34766340139746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Industrial recommender systems critically depend on high-quality ranking models. However, traditional pipelines still rely on manual feature engineering and scenario-specific architectures, which hinder cross-scenario transfer and large-scale deployment. To address these challenges, we propose \textbf{MTmixAtt}, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention, designed for large-scale recommendation tasks. MTmixAtt integrates two key components. The \textbf{AutoToken} module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups. The \textbf{MTmixAttBlock} module enables efficient token interaction via a learnable mixing matrix, shared dense experts, and scenario-aware sparse experts, capturing both global patterns and scenario-specific behaviors within a single framework. Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines including Transformer-based models, WuKong, HiFormer, MLP-Mixer, and RankMixer. At comparable parameter scales, MTmixAtt achieves superior CTR and CTCVR metrics; scaling to MTmixAtt-1B yields further monotonic gains. Large-scale online A/B tests validate the real-world impact: in the \textit{Homepage} scenario, MTmixAtt increases Payment PV by \textbf{+3.62\%} and Actual Payment GTV by \textbf{+2.54\%}. Overall, MTmixAtt provides a unified and scalable solution for modeling arbitrary heterogeneous features across scenarios, significantly improving both user experience and commercial outcomes.

Related papers

• Stable Global Weighting of Flow Mixtures using Simplex Exponential Moving Average [0.8224695424591678]
  emphAMFmbox-VImbox-sEMA is a two-stage framework featuring a emphstable global weighting mechanism.<n>It achieves consistent NLL improvements over its predecessor emphAMFmbox-VI and avoids the catastrophic transport failures of single-flow baselines.
  arXiv  Detail & Related papers  (2026-07-04T10:31:10Z)
• DAG-MoE: From Simple Mixture to Structural Aggregation in Mixture-of-Experts [56.175408382091796]
  We show that replacing the standard weighted-summation aggregation with structural aggregation expands the expert-combination space without altering the experts or router.<n>We propose DAG-MoE, a sparse MoE framework that employs a lightweight module to automatically learn the optimal aggregation structure among the selected experts.
  arXiv  Detail & Related papers  (2026-05-31T07:08:16Z)
• Query-Mixed Interest Extraction and Heterogeneous Interaction: A Scalable CTR Model for Industrial Recommender Systems [6.312847671238921]
  HeMix is a scalable ranking model that unifies adaptive sequence tokenization and heterogeneous interaction structure.<n>HeMix is deployed on the AMAP platform, delivering significant online gains over DLRM: +3.61% GMV, +2.78% PV_CTR, and +2.12% UV_CVR.
  arXiv  Detail & Related papers  (2026-02-10T03:56:14Z)
• MaD-Mix: Multi-Modal Data Mixtures via Latent Space Coupling for Vision-Language Model Training [54.78779514101305]
  MaD-Mix is a principled framework that derives multi-modal data mixtures for VLM training.<n>MaD-Mix speeds VLM training across diverse benchmarks.<n>In complex tri-modal video-image-text scenarios, MaD-Mix boosts average accuracy over uniform weights, with negligible mixture overhead.
  arXiv  Detail & Related papers  (2026-02-08T03:07:36Z)
• MergeMix: Optimizing Mid-Training Data Mixtures via Learnable Model Merging [72.00014675808228]
  textbfMix determines optimal data mixing ratios by repurposing model merging weights as a high-fidelity, low-cost performance proxy.<n>Experiments on models with 8B and 16B parameters validate that MergeMix achieves performance comparable to or surpassing exhaustive manual tuning.
  arXiv  Detail & Related papers  (2026-01-25T14:31:57Z)
• AdaMixT: Adaptive Weighted Mixture of Multi-Scale Expert Transformers for Time Series Forecasting [15.522567372502762]
  We propose a novel architecture named Adaptive Weighted Mixture of Multi-Scale Expert Transformers (AdaMixT)<n>AdaMixT introduces various patches and leverages both General Pre-trained Models (GPM) and Domain-specific Models (DSM) for multi-scale feature extraction.<n> Comprehensive experiments on eight widely used benchmarks, including Weather, Traffic, Electricity, ILI, and four ETT datasets, consistently demonstrate the effectiveness of AdaMixT.
  arXiv  Detail & Related papers  (2025-09-09T15:30:53Z)
• Merge to Mix: Mixing Datasets via Model Merging [2.990932417718553]
  Mixing datasets for fine-tuning large models (LMs) has become critical for maximizing performance on downstream tasks.<n>We propose a novel method, $textitMerge to Mix$, that accelerates composing dataset mixtures through model merging.<n>Our experiments demonstrate that Merge to Mix surpasses state-of-the-art methods in dataset selection for fine-tuning LMs.
  arXiv  Detail & Related papers  (2025-05-21T22:34:13Z)
• PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis [71.8946280170493]
  This paper introduces PowMix, a versatile embedding space regularizer that builds upon the strengths of unimodal mixing-based regularization approaches. PowMix is integrated before the fusion stage of multimodal architectures and facilitates intra-modal mixing, such as mixing text with text, to act as a regularizer.
  arXiv  Detail & Related papers  (2023-12-19T17:01:58Z)
• Adversarial AutoMixup [50.1874436169571]
  We propose AdAutomixup, an adversarial automatic mixup augmentation approach. It generates challenging samples to train a robust classifier for image classification. Our approach outperforms the state of the art in various classification scenarios.
  arXiv  Detail & Related papers  (2023-12-19T08:55:00Z)
• SMMix: Self-Motivated Image Mixing for Vision Transformers [65.809376136455]
  CutMix is a vital augmentation strategy that determines the performance and generalization ability of vision transformers (ViTs) Existing CutMix variants tackle this problem by generating more consistent mixed images or more precise mixed labels. We propose an efficient and effective Self-Motivated image Mixing method (SMMix) which motivates both image and label enhancement by the model under training itself.
  arXiv  Detail & Related papers  (2022-12-26T00:19:39Z)
• Harnessing Hard Mixed Samples with Decoupled Regularizer [69.98746081734441]
  Mixup is an efficient data augmentation approach that improves the generalization of neural networks by smoothing the decision boundary with mixed data. In this paper, we propose an efficient mixup objective function with a decoupled regularizer named Decoupled Mixup (DM) DM can adaptively utilize hard mixed samples to mine discriminative features without losing the original smoothness of mixup.
  arXiv  Detail & Related papers  (2022-03-21T07:12:18Z)
• k-Mixup Regularization for Deep Learning via Optimal Transport [32.951696405505686]
  Mixup is a popular regularization technique for training deep neural networks. We extend mixup in a simple, broadly applicable way to emph$k$-mixup, which perturbs $k$-batches of training points in the direction of other $k$-batches. We show that training with $k$-mixup further improves generalization and robustness across several network architectures.
  arXiv  Detail & Related papers  (2021-06-05T17:08:08Z)
• AutoMix: Unveiling the Power of Mixup [34.623943038648164]
  We present a flexible, general Automatic Mixup framework which utilizes discriminative features to learn a sample mixing policy adaptively. We regard mixup as a pretext task and split it into two sub-problems: mixed samples generation and mixup classification. Experiments on six popular classification benchmarks show that AutoMix consistently outperforms other leading mixup methods.
  arXiv  Detail & Related papers  (2021-03-24T07:21:53Z)
• Patch-level Neighborhood Interpolation: A General and Effective Graph-based Regularization Strategy [77.34280933613226]
  We propose a general regularizer called textbfPatch-level Neighborhood Interpolation(Pani) that conducts a non-local representation in the computation of networks. Our proposal explicitly constructs patch-level graphs in different layers and then linearly interpolates neighborhood patch features, serving as a general and effective regularization strategy.
  arXiv  Detail & Related papers  (2019-11-21T06:31:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.