The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders

• URL: http://arxiv.org/abs/2606.21911v1
• Date: Sat, 20 Jun 2026 07:13:49 GMT
• Title: The Pitfall of Scaling Up: Uncovering and Mitigating Popularity Bias Amplification in Scaling Transformer-based Recommenders
• Abstract summary: We identify a critical pitfall in scaling transformer-based sequential recommenders.<n>While increasing model size improves recommendation accuracy, it simultaneously amplifies popularity bias.<n>This bias drives systems to over-recommend popular items at the expense of niche ones.
• Score: 26.25851138178879
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: We identify a critical pitfall in scaling transformer-based sequential recommenders: while increasing model size improves recommendation accuracy, it simultaneously amplifies popularity bias. This bias drives systems to over-recommend popular items at the expense of niche ones, which not only undermines fairness but also degrades the broader ecosystem by reinforcing the Matthew effect and filter bubbles. Consequently, this bias amplification emerges as a fundamental obstacle to sustainable model scaling. Through comprehensive theoretical and empirical analyses, we uncover the root cause of this amplification. Our findings reveal that as model depth increases, the two core components of the transformer architecture, i.e., attention aggregation and feed-forward projections, synergistically induce severe spectral collapse in model predictions, which directly translates to the amplification of popularity bias. To address this challenge, we propose SPRINT (Scalable Popularity Regularization IN Transformers), which mitigates spectral collapse during scaling by constraining (i) the maximum column-sums of the attention score matrices and (ii) the spectral norms of the feed-forward parameters. Extensive experiments demonstrate that SPRINT significantly improves both accuracy and long-tail fairness. Crucially, it yields more favorable scaling behaviors when expanding model sizes from 0.05M to 0.34B parameters. The code is available at https://github.com/Tiny-Snow/GenRec.

Related papers

• Hierarchical Exponential-Gaussian Mixtures for Watch-Time Distribution Prediction [74.05279072166697]
  Exponential-Gaussian Mixture Network (EGMN) models the full conditional WT distribution.<n>EGMN is vulnerable to variance collapse, component redundancy, and inactive components.<n>We propose a Hierarchical Exponential-Gaussian Mixture (HEGM) model that addresses these failure modes.
  arXiv  Detail & Related papers  (2026-08-24T15:06:18Z)
• OCP: Orthogonal Constrained Projection for Sparse Scaling in Industrial Commodity Recommendation [9.265389086960656]
  A key challenge is that Item-Id vocabularies suffer from low-frequency information interference when subjected to sparse scaling.<n>We propose an Orthogonal Constrained Projection method to optimize embedding representation.
  arXiv  Detail & Related papers  (2026-03-19T09:57:26Z)
• Stabilizing Reinforcement Learning for Diffusion Language Models [27.22260417844743]
  Group Relative Policy Optimization (GRPO) is highly effective for post-training autoregressive (AR) language models.<n>But its direct application to diffusion large language models (dLLMs) often triggers reward collapse.<n>We show these effects form a self-reinforcing instability loop that drives policy drift and further increases ratio variance.<n>We propose StableDRL, a reformulation of GRPO tailored for dLLMs.
  arXiv  Detail & Related papers  (2026-03-06T08:55:37Z)
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training [86.64715217940274]
  Outliers function jointly with normalization.<n>Outliers serve more as rescale factors rather than contributors.<n>Outliers can be absorbed into learnable parameters or mitigated via explicit gated rescaling.
  arXiv  Detail & Related papers  (2026-01-30T13:29:45Z)
• From Scaling to Structured Expressivity: Rethinking Transformers for CTR Prediction [14.997545091069894]
  Deep models for click-through rate (CTR) prediction often exhibit rapidly diminishing returns.<n>We identify the root cause as a structural misalignment.<n>We introduce the Field-Aware Transformer (FAT), which embeds field-based interaction priors into attention.
  arXiv  Detail & Related papers  (2025-11-15T07:55:50Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• Unlocking Noise-Resistant Vision: Key Architectural Secrets for Robust Models [41.429437803093485]
  We investigate why certain vision architectures are inherently more robust to additive Gaussian noise.<n>Specifically, we perform evaluations on 1,174 pretrained vision models.<n>We identify four consistent design patterns for improved robustness against Gaussian noise.
  arXiv  Detail & Related papers  (2025-09-25T09:24:49Z)
• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free [81.65559031466452]
  We conduct experiments to investigate gating-augmented softmax attention variants.<n>We find that a simple modification-applying a head-specific sigmoid gate after the Scaled Dot-Product Attention (SDPA)-consistently improves performance.
  arXiv  Detail & Related papers  (2025-05-10T17:15:49Z)
• How Do Recommendation Models Amplify Popularity Bias? An Analysis from the Spectral Perspective [26.949692163589226]
  Recommendation Systems (RS) are often plagued by popularity bias.<n>This study conducts comprehensive empirical and theoretical analyses to expose the root causes of this phenomenon.<n>We propose a novel debiasing strategy that leverages a spectral norm regularizer to penalize the magnitude of the principal singular value.
  arXiv  Detail & Related papers  (2024-04-18T08:59:32Z)
• Bias in Pruned Vision Models: In-Depth Analysis and Countermeasures [93.17009514112702]
  Pruning, setting a significant subset of the parameters of a neural network to zero, is one of the most popular methods of model compression. Despite existing evidence for this phenomenon, the relationship between neural network pruning and induced bias is not well-understood.
  arXiv  Detail & Related papers  (2023-04-25T07:42:06Z)
• ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
  We propose a new regularizer, namely, Equivariance Regularizer (ER) ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities. The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
  arXiv  Detail & Related papers  (2022-06-24T08:18:05Z)
• PCPL: Predicate-Correlation Perception Learning for Unbiased Scene Graph Generation [58.98802062945709]
  We propose a novel Predicate-Correlation Perception Learning scheme to adaptively seek out appropriate loss weights. Our PCPL framework is further equipped with a graph encoder module to better extract context features.
  arXiv  Detail & Related papers  (2020-09-02T08:30:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.