UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems

• URL: http://arxiv.org/abs/2604.00590v2
• Date: Thu, 02 Apr 2026 02:32:28 GMT
• Title: UniMixer: A Unified Architecture for Scaling Laws in Recommendation Systems
• Abstract summary: We propose a unified scaling architecture for recommendation systems, namely textbfUniMixer, to improve scaling efficiency.<n>We establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods.
• Score: 22.268247447161134
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, the scaling laws of recommendation models have attracted increasing attention, which govern the relationship between performance and parameters/FLOPs of recommenders. Currently, there are three mainstream architectures for achieving scaling in recommendation models, namely attention-based, TokenMixer-based, and factorization-machine-based methods, which exhibit fundamental differences in both design philosophy and architectural structure. In this paper, we propose a unified scaling architecture for recommendation systems, namely \textbf{UniMixer}, to improve scaling efficiency and establish a unified theoretical framework that unifies the mainstream scaling blocks. By transforming the rule-based TokenMixer to an equivalent parameterized structure, we construct a generalized parameterized feature mixing module that allows the token mixing patterns to be optimized and learned during model training. Meanwhile, the generalized parameterized token mixing removes the constraint in TokenMixer that requires the number of heads to be equal to the number of tokens. Furthermore, we establish a unified scaling module design framework for recommender systems, which bridges the connections among attention-based, TokenMixer-based, and factorization-machine-based methods. To further boost scaling ROI, a lightweight UniMixing module is designed, \textbf{UniMixing-Lite}, which further compresses the model parameters and computational cost while significantly improve the model performance. The scaling curves are shown in the following figure. Extensive offline and online experiments are conducted to verify the superior scaling abilities of \textbf{UniMixer}.

Related papers

• Hyperbolic Hierarchical Clustering for Visual Representation Learning [62.38996677789715]
  An effective token mixer is a fundamental component of modern vision backbones like Transformers.<n>We introduce ClusterMixer, a transparent token mixer grounded in a clustering paradigm and interpretable by design.<n>We present HCFormer, a new backbone architecture that integrates ClusterMixer with a series of meticulously designed clustering strategies.
  arXiv  Detail & Related papers  (2026-08-24T00:01:09Z)
• Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation [21.08394922908602]
  Scaling recommendation models is a central challenge in recommender systems.<n>RankMixer suffers from textitembedding collapse, where learned representations have low effective rank.<n>We propose RankElastor, a novel architecture that produces spectrum-robust representations with provable collapse mitigation.
  arXiv  Detail & Related papers  (2026-05-22T03:17:29Z)
• Replacing Parameters with Preferences: Federated Alignment of Heterogeneous Vision-Language Models [63.70401095689976]
  Vision-Language Models (VLMs) have broad potential in privacy-sensitive domains such as healthcare and finance.<n>We propose MoR, a federated alignment framework that combines GRPO with Mixture-of-Rewards for heterogeneousVLMs.<n>MoR consistently outperforms federated alignment baselines in generalization and cross-client adaptability.
  arXiv  Detail & Related papers  (2026-05-05T07:02:50Z)
• MixFormer: Co-Scaling Up Dense and Sequence in Industrial Recommenders [11.566232697512879]
  MixFormer is a unified Transformer-style architecture tailored for recommender systems.<n>It jointly models sequential behaviors and feature interactions within a single backbone.<n>Experiments on large-scale industrial datasets demonstrate that MixFormer consistently exhibits superior accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-02-15T11:53:30Z)
• TokenMixer-Large: Scaling Up Large Ranking Models in Industrial Recommenders [28.610671210049247]
  We propose TokenMixer-Large, a systematically evolved architecture designed for extreme-scale recommendation.<n>By introducing a mixing-and-reverting operation, inter-layer residuals and the auxiliary loss, we ensure stable gradient propagation.<n> TokenMixer-Large successfully scales its parameters to 7-billion and 15-billion on online traffic and offline experiments.
  arXiv  Detail & Related papers  (2026-02-06T10:04:33Z)
• MTmixAtt: Integrating Mixture-of-Experts with Multi-Mix Attention for Large-Scale Recommendation [8.34766340139746]
  We propose textbfMTmixAtt, a unified Mixture-of-Experts (MoE) architecture with Multi-Mix Attention.<n>The textbfAutoToken module automatically clusters heterogeneous features into semantically coherent tokens, removing the need for human-defined feature groups.<n>Extensive experiments on the industrial TRec dataset from Meituan demonstrate that MTmixAtt consistently outperforms state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-10-17T03:50:09Z)
• RankMixer: Scaling Up Ranking Models in Industrial Recommenders [25.072128027628104]
  Training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands.<n>We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture.<n> Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset.
  arXiv  Detail & Related papers  (2025-07-21T12:28:55Z)
• Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
  We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models. Our approach employs activation sparsity to extract experts. Read-ME outperforms other popular open-source dense models of similar scales.
  arXiv  Detail & Related papers  (2024-10-24T19:48:51Z)
• Scaling Pre-trained Language Models to Deeper via Parameter-efficient Architecture [68.13678918660872]
  We design a more capable parameter-sharing architecture based on matrix product operator (MPO) MPO decomposition can reorganize and factorize the information of a parameter matrix into two parts. Our architecture shares the central tensor across all layers for reducing the model size.
  arXiv  Detail & Related papers  (2023-03-27T02:34:09Z)
• Squeezeformer: An Efficient Transformer for Automatic Speech Recognition [99.349598600887]
  Conformer is the de facto backbone model for various downstream speech tasks based on its hybrid attention-convolution architecture. We propose the Squeezeformer model, which consistently outperforms the state-of-the-art ASR models under the same training schemes.
  arXiv  Detail & Related papers  (2022-06-02T06:06:29Z)
• Learning Augmentation Distributions using Transformed Risk Minimization [47.236227685707526]
  We propose a new emphTransformed Risk Minimization (TRM) framework as an extension of classical risk minimization. As a key application, we focus on learning augmentations to improve classification performance with a given class of predictors.
  arXiv  Detail & Related papers  (2021-11-16T02:07:20Z)
• Bayesian Attention Modules [65.52970388117923]
  We propose a scalable version of attention that is easy to implement and optimize. Our experiments show the proposed method brings consistent improvements over the corresponding baselines.
  arXiv  Detail & Related papers  (2020-10-20T20:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.