Lightweight Self-Attentive Sequential Recommendation

• URL: http://arxiv.org/abs/2108.11333v1
• Date: Wed, 25 Aug 2021 16:46:47 GMT
• Title: Lightweight Self-Attentive Sequential Recommendation
• Authors: Yang Li, Tong Chen, Peng-Fei Zhang, Hongzhi Yin
• Abstract summary: We introduce a novel lightweight self-attentive network (LSAN) for sequential recommendation. To aggressively compress the original embedding matrix, LSAN leverages the notion of compositional embeddings. To account for the intrinsic dynamics of each item, we propose a temporal context-aware embedding composition scheme.
• Score: 30.048184102259494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern deep neural networks (DNNs) have greatly facilitated the development of sequential recommender systems by achieving state-of-the-art recommendation performance on various sequential recommendation tasks. Given a sequence of interacted items, existing DNN-based sequential recommenders commonly embed each item into a unique vector to support subsequent computations of the user interest. However, due to the potentially large number of items, the over-parameterised item embedding matrix of a sequential recommender has become a memory bottleneck for efficient deployment in resource-constrained environments, e.g., smartphones and other edge devices. Furthermore, we observe that the widely-used multi-head self-attention, though being effective in modelling sequential dependencies among items, heavily relies on redundant attention units to fully capture both global and local item-item transition patterns within a sequence. In this paper, we introduce a novel lightweight self-attentive network (LSAN) for sequential recommendation. To aggressively compress the original embedding matrix, LSAN leverages the notion of compositional embeddings, where each item embedding is composed by merging a group of selected base embedding vectors derived from substantially smaller embedding matrices. Meanwhile, to account for the intrinsic dynamics of each item, we further propose a temporal context-aware embedding composition scheme. Besides, we develop an innovative twin-attention network that alleviates the redundancy of the traditional multi-head self-attention while retaining full capacity for capturing long- and short-term (i.e., global and local) item dependencies. Comprehensive experiments demonstrate that LSAN significantly advances the accuracy and memory efficiency of existing sequential recommenders.

Related papers

• A Novel Mamba-based Sequential Recommendation Method [4.941272356564765]
  Sequential recommendation (SR) encodes user activity to predict the next action. Transformer-based models have proven effective for sequential recommendation, but the complexity of the self-attention module in Transformers scales quadratically with the sequence length. We propose a novel multi-head latent Mamba architecture, which employs multiple low-dimensional Mamba layers and fully connected layers.
  arXiv  Detail & Related papers  (2025-04-10T02:43:19Z)
• Universal Item Tokenization for Transferable Generative Recommendation [89.42584009980676]
  We propose UTGRec, a universal item tokenization approach for transferable Generative Recommendation. By devising tree-structured codebooks, we discretize content representations into corresponding codes for item tokenization. For raw content reconstruction, we employ dual lightweight decoders to reconstruct item text and images from discrete representations. For collaborative knowledge integration, we assume that co-occurring items are similar and integrate collaborative signals through co-occurrence alignment and reconstruction.
  arXiv  Detail & Related papers  (2025-04-06T08:07:49Z)
• Multi Activity Sequence Alignment via Implicit Clustering [50.3168866743067]
  We propose a novel framework that overcomes limitations using sequence alignment via implicit clustering. Specifically, our key idea is to perform implicit clip-level clustering while aligning frames in sequences. Our experiments show that our proposed method outperforms state-of-the-art results.
  arXiv  Detail & Related papers  (2025-03-16T14:28:46Z)
• Semantic Gaussian Mixture Variational Autoencoder for Sequential Recommendation [49.492451800322144]
  We propose a novel VAE-based Sequential Recommendation model named SIGMA. For multi-interest elicitation, SIGMA includes a probabilistic multi-interest extraction module. Experiments on public datasets demonstrate the effectiveness of SIGMA.
  arXiv  Detail & Related papers  (2025-02-22T08:29:52Z)
• CADMR: Cross-Attention and Disentangled Learning for Multimodal Recommender Systems [0.6037276428689637]
  We propose CADMR, a novel autoencoder-based multimodal recommender system framework. We evaluate CADMR on three benchmark datasets, demonstrating significant performance improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-12-03T09:09:52Z)
• Unleash LLMs Potential for Recommendation by Coordinating Twin-Tower Dynamic Semantic Token Generator [60.07198935747619]
  We propose Twin-Tower Dynamic Semantic Recommender (T TDS), the first generative RS which adopts dynamic semantic index paradigm. To be more specific, we for the first time contrive a dynamic knowledge fusion framework which integrates a twin-tower semantic token generator into the LLM-based recommender. The proposed T TDS recommender achieves an average improvement of 19.41% in Hit-Rate and 20.84% in NDCG metric, compared with the leading baseline methods.
  arXiv  Detail & Related papers  (2024-09-14T01:45:04Z)
• GLINT-RU: Gated Lightweight Intelligent Recurrent Units for Sequential Recommender Systems [24.758106816702767]
  This paper proposes a novel efficient sequential recommendation framework GLINT-RU. It uses dense selective Gated Recurrent Units (GRU) module to accelerate the inference speed. Our framework achieves exceptional inference speed and prediction accuracy, outperforming existing baselines.
  arXiv  Detail & Related papers  (2024-06-06T13:55:55Z)
• MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation [61.45986275328629]
  We propose MISSRec, a multi-modal pre-training and transfer learning framework for sequential recommendation. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal user interests. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation.
  arXiv  Detail & Related papers  (2023-08-22T04:06:56Z)
• Enhancing Few-shot NER with Prompt Ordering based Data Augmentation [59.69108119752584]
  We propose a Prompt Ordering based Data Augmentation (PODA) method to improve the training of unified autoregressive generation frameworks. Experimental results on three public NER datasets and further analyses demonstrate the effectiveness of our approach.
  arXiv  Detail & Related papers  (2023-05-19T16:25:43Z)
• Generative Slate Recommendation with Reinforcement Learning [49.75985313698214]
  reinforcement learning algorithms can be used to optimize user engagement in recommender systems. However, RL approaches are intractable in the slate recommendation scenario. In that setting, an action corresponds to a slate that may contain any combination of items. In this work we propose to encode slates in a continuous, low-dimensional latent space learned by a variational auto-encoder. We are able to (i) relax assumptions required by previous work, and (ii) improve the quality of the action selection by modeling full slates.
  arXiv  Detail & Related papers  (2023-01-20T15:28:09Z)
• Multi-Behavior Hypergraph-Enhanced Transformer for Sequential Recommendation [33.97708796846252]
  We introduce a new Multi-Behavior Hypergraph-enhanced Transformer framework (MBHT) to capture both short-term and long-term cross-type behavior dependencies. Specifically, a multi-scale Transformer is equipped with low-rank self-attention to jointly encode behavior-aware sequential patterns from fine-grained and coarse-grained levels.
  arXiv  Detail & Related papers  (2022-07-12T15:07:21Z)
• From STL Rulebooks to Rewards [4.859570041295978]
  We propose a principled approach to shaping rewards for reinforcement learning from multiple objectives. We first equip STL with a novel quantitative semantics allowing to automatically evaluate individual requirements. We then develop a method for systematically combining evaluations of multiple requirements into a single reward.
  arXiv  Detail & Related papers  (2021-10-06T14:16:59Z)
• Sparse-Interest Network for Sequential Recommendation [78.83064567614656]
  We propose a novel textbfSparse textbfInterest textbfNEtwork (SINE) for sequential recommendation. Our sparse-interest module can adaptively infer a sparse set of concepts for each user from the large concept pool. SINE can achieve substantial improvement over state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-02-18T11:03:48Z)
• Sequential Recommendation with Self-Attentive Multi-Adversarial Network [101.25533520688654]
  We present a Multi-Factor Generative Adversarial Network (MFGAN) for explicitly modeling the effect of context information on sequential recommendation. Our framework is flexible to incorporate multiple kinds of factor information, and is able to trace how each factor contributes to the recommendation decision over time.
  arXiv  Detail & Related papers  (2020-05-21T12:28:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.