RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems

• URL: http://arxiv.org/abs/2604.17878v2
• Date: Tue, 21 Apr 2026 03:44:53 GMT
• Title: RankUp: Towards High-rank Representations for Large Scale Advertising Recommender Systems
• Abstract summary: We propose textbfRankUp, an architecture designed to mitigate representation collapse and enhance expressive capacity.<n>RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41%, 4.81% and 2.21%, respectively.
• Score: 21.830994984868212
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The scaling laws for recommender systems have been increasingly validated, where MetaFormer-based architectures consistently benefit from increased model depth, hidden dimensionality, and user behavior sequence length. However, whether representation capacity scales proportionally with parameter growth remains largely unexplored. Prior studies on RankMixer reveal that the effective rank of token representations exhibits a damped oscillatory trajectory across layers, failing to increase consistently with depth and even degrading in deeper layers. Motivated by this observation, we propose \textbf{RankUp}, an architecture designed to mitigate representation collapse and enhance expressive capacity through randomized permutation splitting over sparse features, a multi-embedding paradigm, global token integration, crossed pretrained embedding tokens and task-specific token decoupling. RankUp has been fully deployed in large-scale production across Weixin Video Accounts, Official Accounts and Moments, yielding GMV improvements of 3.41\%, 4.81\% and 2.21\%, respectively.

Related papers

• Collaborative Memory Augmentation for Generative Recommendation [49.76630658239287]
  We propose textbfOMEGA, a cOllaborative MEmory augmentation framework for Generative recommendAtion.<n>We first introduce a latent context compression method that utilizes learnable query tokens to distill sequential user behavior into compact representations.<n>These compressed representations are aggregated into a collaborative memory bank, serving as an explicit repository of global behavioral patterns.
  arXiv  Detail & Related papers  (2026-08-02T15:34:43Z)
• SlimPer: Make Personalization Model Slim and Smart [20.68857468213109]
  We propose SlimPer to reformulate personalized ranking as iterative refinement of a compact, unified user, item> knowledge base.<n>SlimPer unifies sparse, dense, and sequence features within a single backbone and provides inherent interpretability through its attention mechanism.
  arXiv  Detail & Related papers  (2026-07-14T02:31:32Z)
• Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation [21.08394922908602]
  Scaling recommendation models is a central challenge in recommender systems.<n>RankMixer suffers from textitembedding collapse, where learned representations have low effective rank.<n>We propose RankElastor, a novel architecture that produces spectrum-robust representations with provable collapse mitigation.
  arXiv  Detail & Related papers  (2026-05-22T03:17:29Z)
• TokenFormer: Unify the Multi-Field and Sequential Recommendation Worlds [17.505701673051405]
  We propose TokenFormer, a unified recommendation architecture with the following innovations.<n>First, we introduce a Bottom-Full-Top-Sliding (BFTS) attention scheme, which applies full self-attention in the lower layers and shrinking-window sliding attention in the upper layers.<n>Second, we introduce a Non-linear Interaction Representation (NLIR) that applies one-sided non-linear multiplicative transformations to the hidden states.
  arXiv  Detail & Related papers  (2026-04-15T11:25:46Z)
• Attention Residuals [38.59138244826294]
  Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights.<n>We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs.
  arXiv  Detail & Related papers  (2026-03-16T09:32:21Z)
• StepVAR: Structure-Texture Guided Pruning for Visual Autoregressive Models [98.72926158261937]
  We propose a training-free token pruning framework for Visual AutoRegressive models.<n>We employ a lightweight high-pass filter to capture local texture details, while leveraging Principal Component Analysis (PCA) to preserve global structural information.<n>To maintain valid next-scale prediction under sparse tokens, we introduce a nearest neighbor feature propagation strategy.
  arXiv  Detail & Related papers  (2026-03-02T11:35:05Z)
• Beyond the Flat Sequence: Hierarchical and Preference-Aware Generative Recommendations [35.58864660038236]
  We propose a novel framework named HPGR (Hierarchical and Preference-aware Generative Recommender)<n>First, a structure-aware pre-training stage employs a session-based Masked Item Modeling objective to learn a hierarchically-informed and semantically rich item representation space.<n>Second, a preference-aware fine-tuning stage leverages these powerful representations to implement a Preference-Guided Sparse Attention mechanism.
  arXiv  Detail & Related papers  (2026-03-01T08:15:34Z)
• Granular-ball Guided Masking: Structure-aware Data Augmentation [97.18560547134587]
  Granular-ball Guided Masking (GBGM) is a structure-aware augmentation strategy guided by Granular-ball Computing (GBC)<n>GBGM adaptively preserves semantically rich, structurally important regions while suppressing redundant areas through a coarse-to-fine hierarchical masking process.<n>Experiments on multiple benchmarks demonstrate consistent improvements in classification accuracy and masked image reconstruction.
  arXiv  Detail & Related papers  (2025-12-24T07:15:33Z)
• MeshMosaic: Scaling Artist Mesh Generation via Local-to-Global Assembly [62.48017648785026]
  We introduce MeshMosaic, a novel local-to-global framework for artist mesh generation that scales to over 100K triangles.<n>We show that MeshMosaic significantly outperforms state-of-the-art methods in both geometric fidelity and user preference.
  arXiv  Detail & Related papers  (2025-09-24T11:02:03Z)
• Contrastive Matrix Completion with Denoising and Augmented Graph Views for Robust Recommendation [1.0128808054306186]
  Matrix completion is a widely adopted framework in recommender systems.<n>We propose a novel method called Matrix Completion using Contrastive Learning (MCCL)<n>Our approach not only improves the numerical accuracy of the predicted scores--but also produces superior rankings with improvements of up to 36% in ranking metrics.
  arXiv  Detail & Related papers  (2025-06-12T12:47:35Z)
• You Do Not Fully Utilize Transformer's Representation Capacity [4.753535328327317]
  Layer-Integrated Memory (LIMe) is a lightweight extension that learns per-head, per-layer routing weights to integrate representations from all previous layers with negligible overhead.<n>LIMe consistently achieves faster convergence, lower perplexity per FLOP, and substantial accuracy improvements on synthetic tasks.
  arXiv  Detail & Related papers  (2025-02-13T12:00:50Z)
• Augmenting Convolutional networks with attention-based aggregation [55.97184767391253]
  We show how to augment any convolutional network with an attention-based global map to achieve non-local reasoning. We plug this learned aggregation layer with a simplistic patch-based convolutional network parametrized by 2 parameters (width and depth) It yields surprisingly competitive trade-offs between accuracy and complexity, in particular in terms of memory consumption.
  arXiv  Detail & Related papers  (2021-12-27T14:05:41Z)
• Deep Autoencoding Topic Model with Scalable Hybrid Bayesian Inference [55.35176938713946]
  We develop deep autoencoding topic model (DATM) that uses a hierarchy of gamma distributions to construct its multi-stochastic-layer generative network. We propose a Weibull upward-downward variational encoder that deterministically propagates information upward via a deep neural network, followed by a downward generative model. The efficacy and scalability of our models are demonstrated on both unsupervised and supervised learning tasks on big corpora.
  arXiv  Detail & Related papers  (2020-06-15T22:22:56Z)
• Crowd Counting via Hierarchical Scale Recalibration Network [61.09833400167511]
  We propose a novel Hierarchical Scale Recalibration Network (HSRNet) to tackle the task of crowd counting. HSRNet models rich contextual dependencies and recalibrating multiple scale-associated information. Our approach can ignore various noises selectively and focus on appropriate crowd scales automatically.
  arXiv  Detail & Related papers  (2020-03-07T10:06:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.