TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders

• URL: http://arxiv.org/abs/2602.06563v2
• Date: Tue, 10 Feb 2026 02:17:46 GMT
• Title: TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders
• Abstract summary: We propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation.<n>By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation.<n> TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments.
• Score: 28.610671210049247
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While scaling laws for recommendation models have gained significant traction, existing architectures such as Wukong, HiFormer and DHEN, often struggle with sub-optimal designs and hardware under-utilization, limiting their practical scalability. Our previous TokenMixer architecture (introduced in RankMixer paper) addressed effectiveness and efficiency by replacing self-attention with a ightweight token-mixing operator; however, it faced critical bottlenecks in deeper configurations, including sub-optimal residual paths, vanishing gradients, incomplete MoE sparsification and constrained scalability. In this paper, we propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation. By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation even as model depth increases. Furthermore, we incorporate a Sparse Per-token MoE to enable efficient parameter expansion. TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments, respectively. Currently deployed in multiple scenarios at ByteDance, TokenMixer-Large has achieved significant offline and online performance gains, delivering an increase of +1.66\% in orders and +2.98\% in per-capita preview payment GMV for e-commerce, improving ADSS by +2.0\% in advertising and achieving a +1.4\% revenue growth for live streaming.

Related papers

• MoEMB: Scaling Universal Multimodal Embeddings with Efficient Mixture-of-Experts Models [55.732137666068446]
  Universal multimodal embedding (UME) increasingly demands encoder's capacity for handling a broad range of tasks and modalities.<n>MoEMB scales UME along the expert axis through mixture-of-experts (MoE)<n>MoEMB surpasses TTE-based methods with >4x active parameters, using significantly less computes.
  arXiv  Detail & Related papers  (2026-09-08T12:29:50Z)
• RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems [21.830994984868212]
  We propose textbfRankUp, an architecture designed to mitigate representation collapse and enhance expressive capacity.<n>RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41%, 4.81% and 2.21%, respectively.
  arXiv  Detail & Related papers  (2026-04-20T06:40:27Z)
• UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems [22.268247447161134]
  We propose a unified scaling architecture for recommendation systems, namely textbfUniMixer, to improve scaling efficiency.<n>We establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods.
  arXiv  Detail & Related papers  (2026-04-01T07:57:40Z)
• Controlled LLM Training on Spectral Sphere [76.60985966206746]
  We introduce the textbfSpectral Sphere algorithm (SSO), which enforces strict module-wise spectral constraints on both weights and their updates.<n>We observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations.
  arXiv  Detail & Related papers  (2026-01-13T09:59:47Z)
• Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Models [97.55009021098554]
  This work aims to identify the key determinants of SLMs' real-device latency and offer generalizable principles and methodologies for SLM design and training.<n>We introduce a new family of hybrid SLMs, called Nemotron-Flash, which significantly advances the accuracy-efficiency frontier of state-of-the-art SLMs.
  arXiv  Detail & Related papers  (2025-11-24T08:46:36Z)
• MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation [8.34766340139746]
  We propose textbfMTmixAtt, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention.<n>The textbfAutoToken module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups.<n>Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-17T03:50:09Z)
• MC#: Mixture Compressor for Mixture-of-Experts Large Models [86.64315380917827]
  Mixture-of-Experts (MoE) effectively scales large language models (LLMs) and vision-language models (VLMs) by increasing capacity through sparse activation.<n>We propose MC# (Mixture-Compressor-sharp), a framework that combines static quantization and dynamic expert pruning.
  arXiv  Detail & Related papers  (2025-10-13T03:12:46Z)
• RankMixer: Scaling Up Ranking Models in Industrial Recommenders [25.072128027628104]
  Training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands.<n>We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture.<n> Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset.
  arXiv  Detail & Related papers  (2025-07-21T12:28:55Z)
• An Efficient and Mixed Heterogeneous Model for Image Restoration [71.85124734060665]
  Current mainstream approaches are based on three architectural paradigms: CNNs, Transformers, and Mambas.<n>We propose RestorMixer, an efficient and general-purpose IR model based on mixed-architecture fusion.
  arXiv  Detail & Related papers  (2025-04-15T08:19:12Z)
• Mixture of Experts in Image Classification: What's the Sweet Spot? [41.72573567802606]
  We study the integration of MoE layers into image classification architectures using open datasets.<n>We find that moderate parameter activation per sample provides the best trade-off between performance and efficiency.<n>MoE layers most effectively strengthen tiny and mid-sized models, while gains taper off for large-capacity networks.
  arXiv  Detail & Related papers  (2024-11-27T13:23:11Z)
• TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters [102.1116808722299]
  We introduce TokenFormer, a scalable architecture for scaling Transformers.<n>By treating model parameters as tokens, we replace all the linear projections in Transformers.<n>Our model scales from 124M to 1.4B parameters by incrementally adding new key-value parameter pairs.
  arXiv  Detail & Related papers  (2024-10-30T16:19:00Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference [13.263938935671646]
  AdapMoE is an algorithm-system co-design framework for efficient MoE inference. AdapMoE features adaptive expert gating and management to reduce the on-demand loading overheads. We show AdapMoE consistently outperforms existing techniques, reducing the average number of activated experts by 25% and achieving a 1.35x speedup without degradation accuracy.
  arXiv  Detail & Related papers  (2024-08-19T03:27:15Z)
• Layerwise Recurrent Router for Mixture-of-Experts [42.36093735411238]
  Mixture-of-Experts (MoE) architecture stands out for its ability to scale model size without significantly increasing training costs.<n>Current MoE models often display parameter inefficiency.<n>We introduce the Layerwise Recurrent Router for Mixture-of-Experts (RMoE)
  arXiv  Detail & Related papers  (2024-08-13T10:25:13Z)
• Mixture of Tokens: Continuous MoE through Cross-Example Aggregation [0.7880651741080428]
  Mixture of Experts (MoE) models are pushing the boundaries of language and vision tasks. MoT is a simple, continuous architecture that is capable of scaling the number of parameters similarly to sparse MoE models. Our best models achieve a 3x increase in training speed over dense Transformer models in language pretraining.
  arXiv  Detail & Related papers  (2023-10-24T16:03:57Z)
• RIFormer: Keep Your Vision Backbone Effective While Removing Token Mixer [95.71132572688143]
  This paper studies how to keep a vision backbone effective while removing token mixers in its basic building blocks. Token mixers, as self-attention for vision transformers (ViTs), are intended to perform information communication between different spatial tokens but suffer from considerable computational cost and latency.
  arXiv  Detail & Related papers  (2023-04-12T07:34:13Z)
• EdgeNeXt: Efficiently Amalgamated CNN-Transformer Architecture for Mobile Vision Applications [68.35683849098105]
  We introduce split depth-wise transpose attention (SDTA) encoder that splits input tensors into multiple channel groups. Our EdgeNeXt model with 1.3M parameters achieves 71.2% top-1 accuracy on ImageNet-1K. Our EdgeNeXt model with 5.6M parameters achieves 79.4% top-1 accuracy on ImageNet-1K.
  arXiv  Detail & Related papers  (2022-06-21T17:59:56Z)
• Parameter-Efficient Mixture-of-Experts Architecture for Pre-trained Language Models [68.9288651177564]
  We present a novel MoE architecture based on matrix product operators (MPO) from quantum many-body physics. With the decomposed MPO structure, we can reduce the parameters of the original MoE architecture. Experiments on the three well-known downstream natural language datasets based on GPT2 show improved performance and efficiency in increasing model capacity.
  arXiv  Detail & Related papers  (2022-03-02T13:44:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.