RankMixer: Scaling Up Ranking Models in Industrial Recommenders

• URL: http://arxiv.org/abs/2507.15551v3
• Date: Sat, 26 Jul 2025 02:01:33 GMT
• Title: RankMixer: Scaling Up Ranking Models in Industrial Recommenders
• Abstract summary: Training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands.<n>We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture.<n> Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset.
• Score: 25.072128027628104
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress on large language models (LLMs) has spurred interest in scaling up recommendation systems, yet two practical obstacles remain. First, training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands. Second, most human-designed feature-crossing modules in ranking models were inherited from the CPU era and fail to exploit modern GPUs, resulting in low Model Flops Utilization (MFU) and poor scalability. We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture. RankMixer retains the transformer's high parallelism while replacing quadratic self-attention with multi-head token mixing module for higher efficiency. Besides, RankMixer maintains both the modeling for distinct feature subspaces and cross-feature-space interactions with Per-token FFNs. We further extend it to one billion parameters with a Sparse-MoE variant for higher ROI. A dynamic routing strategy is adapted to address the inadequacy and imbalance of experts training. Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset. By replacing previously diverse handcrafted low-MFU modules with RankMixer, we boost the model MFU from 4.5\% to 45\%, and scale our ranking model parameters by 100x while maintaining roughly the same inference latency. We verify RankMixer's universality with online A/B tests across two core application scenarios (Recommendation and Advertisement). Finally, we launch 1B Dense-Parameters RankMixer for full traffic serving without increasing the serving cost, which improves user active days by 0.3\% and total in-app usage duration by 1.08\%.

Related papers

• Temporally Extended Mixture-of-Experts Models [6.07059176899521]
  Mixture-of-Experts models are popular for scaling capacity at fixed inference speed.<n>Once a model outgrows available GPU memory, this churn can render optimizations like offloading and pre-fetching ineffective.<n>We make the case that the options framework in reinforcement learning is a perfect match to tackle this problem, and argue for temporally extended mixture-of-experts layers.
  arXiv  Detail & Related papers  (2026-04-22T03:50:52Z)
• UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems [22.268247447161134]
  We propose a unified scaling architecture for recommendation systems, namely textbfUniMixer, to improve scaling efficiency.<n>We establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods.
  arXiv  Detail & Related papers  (2026-04-01T07:57:40Z)
• SEMixer: Semantics Enhanced MLP-Mixer for Multiscale Mixing and Long-term Time Series Forecasting [9.398256560898448]
  SEMixer is a lightweight model designed for long-term time series forecasting (TSF)<n>Mixer features a Random Attention Mechanism (RAM) and a Multiscale Progressive Mixing Chain (MPMC)<n>MPMC stacks RAM and ensemble-Mixer in a memory-efficient manner, achieving more effective temporal mixing.
  arXiv  Detail & Related papers  (2026-02-18T06:53:32Z)
• Query-Mixed Interest Extraction and Heterogeneous Interaction: A Scalable CTR Model for Industrial Recommender Systems [6.312847671238921]
  HeMix is a scalable ranking model that unifies adaptive sequence tokenization and heterogeneous interaction structure.<n>HeMix is deployed on the AMAP platform, delivering significant online gains over DLRM: +3.61% GMV, +2.78% PV_CTR, and +2.12% UV_CVR.
  arXiv  Detail & Related papers  (2026-02-10T03:56:14Z)
• TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders [28.610671210049247]
  We propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation.<n>By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation.<n> TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments.
  arXiv  Detail & Related papers  (2026-02-06T10:04:33Z)
• Diffusion Language Models are Super Data Learners [61.721441061210896]
  When unique data is limited, diffusion language models (DLMs) consistently surpass autoregressive (AR) models by training for more epochs.<n>We attribute the gains to three compounding factors: (1) any-order modeling, (2) super-dense compute from iterative bidirectional denoising, and (3) built-in Monte Carlo augmentation.
  arXiv  Detail & Related papers  (2025-11-05T08:17:42Z)
• MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation [8.34766340139746]
  We propose textbfMTmixAtt, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention.<n>The textbfAutoToken module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups.<n>Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-17T03:50:09Z)
• MoEs Are Stronger than You Think: Hyper-Parallel Inference Scaling with RoE [12.96406947372715]
  We introduce hyper-parallel scaling, a complementary framework that improves prediction quality at the token level.<n>We implement this concept in Mixture-of-Experts (MoE) models, which we refer to as Roster of Experts (RoE)<n>RoE is a training-free inference algorithm that turns a single MoE into a dynamic ensemble of MoEs.
  arXiv  Detail & Related papers  (2025-09-21T21:05:29Z)
• SPIRE: Conditional Personalization for Federated Diffusion Generative Models [7.8583640700306585]
  Shared Backbone Personal Identity Representation Embeddings (SPIRE) is a framework that casts per client diffusion based generation as conditional generation in FL.<n>SPIRE factorizes the network into (i) a high capacity global backbone that learns a population level score function and (ii) lightweight, learnable client embeddings that encode local data statistics.<n>Our analysis also hints at how client embeddings act as biases that steer a shared score network toward personalized distributions.
  arXiv  Detail & Related papers  (2025-06-14T01:40:31Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• RegMix: Data Mixture as Regression for Language Model Pre-training [40.45464495981735]
  We propose RegMix to automatically identify a high-performing data mixture by formulating it as a regression task.<n>RegMix trains many small models on diverse data mixtures, uses regression to predict performance of unseen mixtures, and applies the best predicted mixture to train a large-scale model with orders of magnitude more compute.
  arXiv  Detail & Related papers  (2024-07-01T17:31:03Z)
• Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training [73.90260246781435]
  We present Lory, the first approach that scales such architectures to autoregressive language model pre-training. We show significant performance gains over parameter-matched dense models on both perplexity and a variety of downstream tasks. Despite segment-level routing, Lory models achieve competitive performance compared to state-of-the-art MoE models with token-level routing.
  arXiv  Detail & Related papers  (2024-05-06T03:06:33Z)
• HiRE: High Recall Approximate Top-$k$ Estimation for Efficient LLM Inference [68.59839755875252]
  HiRE comprises of two novel components: (i) a compression scheme to cheaply predict top-$k$ rows/columns with high recall, followed by full computation restricted to the predicted subset, and (ii) DA-TOP-$k$: an efficient multi-device approximate top-$k$ operator. We demonstrate that on a one billion parameter model, HiRE applied to both the softmax as well as feedforward layers, achieves almost matching pretraining and downstream accuracy, and speeds up inference latency by $1.47times$ on a single TPUv5e device.
  arXiv  Detail & Related papers  (2024-02-14T18:04:36Z)
• AutoMix: Automatically Mixing Language Models [62.51238143437967]
  Large language models (LLMs) are now available from cloud API providers in various sizes and configurations.<n>We present Automix, an approach that strategically routes queries to larger LMs, based on the approximate correctness of outputs from a smaller LM.
  arXiv  Detail & Related papers  (2023-10-19T17:57:39Z)
• TSMixer: Lightweight MLP-Mixer Model for Multivariate Time Series Forecasting [13.410217680999459]
  Transformers have gained popularity in time series forecasting for their ability to capture long-sequence interactions. High memory and computing requirements pose a critical bottleneck for long-term forecasting. We propose TSMixer, a lightweight neural architecture composed of multi-layer perceptron (MLP) modules.
  arXiv  Detail & Related papers  (2023-06-14T06:26:23Z)
• AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuning [112.97430455461097]
  We propose a general PEFT method that tunes a mixture of adaptation modules introduced in each Transformer layer while keeping most of the PLM weights frozen. By only tuning 0.1-0.2% of PLM parameters, we show that AdaMix outperforms SOTA parameter-efficient fine-tuning and full model fine-tuning for both NLU and NLG tasks.
  arXiv  Detail & Related papers  (2022-10-31T16:23:36Z)
• Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference [17.97893143555333]
  Sparse Mixture-of-Experts (MoE) has been a successful approach for scaling multilingual translation models to billions of parameters without a proportional increase in training computation. In this work, we investigate routing strategies at different granularity (token, sentence, task) in MoE models to bypass distillation. Experiments on WMT and a web-scale dataset suggest that task-level routing (task-MoE) enables us to extract smaller, ready-to-deploy sub-networks from large sparse models.
  arXiv  Detail & Related papers  (2021-09-24T20:42:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.