Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation

• URL: http://arxiv.org/abs/2604.20858v1
• Date: Sun, 01 Mar 2026 23:20:48 GMT
• Title: Mixture of Sequence: Theme-Aware Mixture-of-Experts for Long-Sequence Recommendation
• Abstract summary: We propose a model-agnostic MoE approach that achieves accurate predictions by extracting theme-specific and multi-scale subsequences from noisy raw user sequences.<n> Experimental results demonstrate that MoS consistently achieves the SOTA performance while introducing fewer FLOPs.
• Score: 48.98369214412951
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Sequential recommendation has rapidly advanced in click-through rate prediction due to its ability to model dynamic user interests. A key challenge, however, lies in modeling long sequences: users often exhibit significant interest shifts, introducing substantial irrelevant or misleading information. Our empirical analysis corroborates this challenge and uncovers a recurring behavioral pattern in long sequences (\textit{session hopping}): user interests remain stable within short temporal spans (\textit{sessions}) but shift drastically across sessions and may reappear after multiple sessions. To address this challenge, we propose the Mixture of Sequence (MoS) framework, a model-agnostic MoE approach that achieves accurate predictions by extracting theme-specific and multi-scale subsequences from noisy raw user sequences. First, MoS employs a theme-aware routing mechanism to adaptively learn the latent themes of user sequences and organizes these sequences into multiple coherent subsequences. Each subsequence contains only sessions aligned with a specific theme, thereby effectively filtering out irrelevant or even misleading information introduced by user interest shifts in session hopping. In addition, to alleviate potential information loss, we introduce a multi-scale fusion mechanism, which leverages three types of experts to capture global sequence characteristics, short-term user behaviors, and theme-specific semantic patterns. Together, these two mechanisms endow MoS with the ability to deliver accurate recommendations from multi-faceted and multi-scale perspectives. Experimental results demonstrate that MoS consistently achieves the SOTA performance while introducing fewer FLOPs compared with other MoE counterparts, providing strong evidence of its excellent balance between utility and efficiency. The code is available at https://github.com/xiaolin-cs/MoS.

Related papers

• Drift-Aware Multimodal User Representation Learning via Multi-Scale Temporal Modeling and Sparse Mixture-of-Experts [8.624467983444255]
  We propose DUMoE, a unified framework for drift-aware multimodal user representation learning.<n>Our model consists of (i) a temporal dynamics-aware backbone that captures and integrates static profiles, short-term behavioral signals, and long-term dependencies into a coherent representation, and (ii) a sparse mixture-of-experts (MoE) interest adapter that disentangles multiple latent interests via expert specialization and adaptive routing.<n>Experiments on real-world social media datasets show that DUMoE consistently outperforms state-of-the-art methods on both user interest prediction and interaction prediction tasks.
  arXiv  Detail & Related papers  (2026-08-26T13:15:17Z)
• GEMs: Breaking the Long-Sequence Barrier in Generative Recommendation with a Multi-Stream Decoder [54.64137490632567]
  We propose a novel and unified framework designed to capture users' sequences from long-term history.<n>Generative Multi-streamers ( GEMs) break user sequences into three streams.<n>Extensive experiments on large-scale industrial datasets demonstrate that GEMs significantly outperforms state-the-art methods in recommendation accuracy.
  arXiv  Detail & Related papers  (2026-02-14T06:42:56Z)
• Unleashing the Potential of Sparse Attention on Long-term Behaviors for CTR Prediction [17.78352301235849]
  We propose SparseCTR, an efficient and effective model specifically designed for long-term behaviors of users.<n>Based on these chunks, we propose a three-branch sparse self-attention mechanism to jointly identify users' global interests.<n>We show that SparseCTR not only improves efficiency but also outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-01-25T13:39:26Z)
• STAR-Rec: Making Peace with Length Variance and Pattern Diversity in Sequential Recommendation [61.320991769685065]
  STAR-Rec is a novel architecture that combines preference-aware attention and state-space modeling.<n>We show that STAR-Rec consistently outperforms state-of-the-art sequential recommendation methods.
  arXiv  Detail & Related papers  (2025-05-06T12:40:38Z)
• Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation [19.47124940518026]
  We propose a Hierarchical time-aware Mixture of experts for multi-modal Sequential Recommendation (HM4SR)<n>First MoE, named Interactive MoE, extracts essential user interest-related information from the multi-modal data of each item.<n>Second MoE, termed Temporal MoE, captures user dynamic interests by introducing explicit temporal embeddings from timestamps in modality encoding.
  arXiv  Detail & Related papers  (2025-01-24T06:26:50Z)
• Multi-granularity Interest Retrieval and Refinement Network for Long-Term User Behavior Modeling in CTR Prediction [68.90783662117936]
  Click-through Rate (CTR) prediction is crucial for online personalization platforms.<n>Recent advancements have shown that modeling rich user behaviors can significantly improve the performance of CTR prediction.<n>We propose Multi-granularity Interest Retrieval and Refinement Network (MIRRN)
  arXiv  Detail & Related papers  (2024-11-22T15:29:05Z)
• MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation [61.45986275328629]
  We propose MISSRec, a multi-modal pre-training and transfer learning framework for sequential recommendation. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal user interests. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation.
  arXiv  Detail & Related papers  (2023-08-22T04:06:56Z)
• Contrastive Self-supervised Sequential Recommendation with Robust Augmentation [101.25762166231904]
  Sequential Recommendationdescribes a set of techniques to model dynamic user behavior in order to predict future interactions in sequential user data. Old and new issues remain, including data-sparsity and noisy data. We propose Contrastive Self-Supervised Learning for sequential Recommendation (CoSeRec)
  arXiv  Detail & Related papers  (2021-08-14T07:15:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.