A Contextual-Aware Position Encoding for Sequential Recommendation

• URL: http://arxiv.org/abs/2502.09027v2
• Date: Sat, 22 Feb 2025 02:06:55 GMT
• Title: A Contextual-Aware Position Encoding for Sequential Recommendation
• Authors: Jun Yuan, Guohao Cai, Zhenhua Dong,
• Abstract summary: Sequential recommendation (SR) encodes user activity to predict the next action.<n>We propose a novel Contextual-Aware Position textual method for sequential recommendation, abbreviated as CAPE.
• Score: 19.338997746519897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sequential recommendation (SR), which encodes user activity to predict the next action, has emerged as a widely adopted strategy in developing commercial personalized recommendation systems. A critical component of modern SR models is the attention mechanism, which synthesizes users' historical activities. This mechanism is typically order-invariant and generally relies on position encoding (PE). Conventional SR models simply assign a learnable vector to each position, resulting in only modest gains compared to traditional recommendation models. Moreover, limited research has been conducted on position encoding tailored for sequential recommendation, leaving a significant gap in addressing its unique requirements. To bridge this gap, we propose a novel Contextual-Aware Position Encoding method for sequential recommendation, abbreviated as CAPE. To the best of our knowledge, CAPE is the first PE method specifically designed for sequential recommendation. Comprehensive experiments conducted on benchmark SR datasets demonstrate that CAPE consistently enhances multiple mainstream backbone models and achieves state-of-the-art performance, across small and large scale model size. Furthermore, we deployed CAPE in an industrial setting on a real-world commercial platform, clearly showcasing the effectiveness of our approach. Our source code is available at https://github.com/yjdy/CAPE.

Related papers

• Spacetime-GR: A Spacetime-Aware Generative Model for Large Scale Online POI Recommendation [31.53029907013095]
  We propose Spacetime-GR, the first spacetime-aware generative model for large-scale online Point-of-Interest (POI) recommendation.<n>It extends the strong sequence modeling ability incorporating generative models by flexible encoding.<n>We evaluate the proposed model on both public benchmark datasets and largescale industrial datasets.
  arXiv  Detail & Related papers  (2025-08-22T06:37:57Z)
• Closing the Performance Gap in Generative Recommenders with Collaborative Tokenization and Efficient Modeling [10.757287948514604]
  We introduce a contrastive tokenization method that integrates collaborative information directly into the learned item representations.<n>We also propose MARIUS, a lightweight, audio-inspired generative model that decouples timeline modeling from item decoding.
  arXiv  Detail & Related papers  (2025-08-12T17:06:55Z)
• LARES: Latent Reasoning for Sequential Recommendation [96.26996622771593]
  We present LARES, a novel and scalable LAtent REasoning framework for Sequential recommendation.<n>Our proposed approach employs a recurrent architecture that allows flexible expansion of reasoning depth without increasing parameter complexity.<n>Our framework exhibits seamless compatibility with existing advanced models, further improving their recommendation performance.
  arXiv  Detail & Related papers  (2025-05-22T16:22:54Z)
• A Novel Mamba-based Sequential Recommendation Method [4.941272356564765]
  Sequential recommendation (SR) encodes user activity to predict the next action. Transformer-based models have proven effective for sequential recommendation, but the complexity of the self-attention module in Transformers scales quadratically with the sequence length. We propose a novel multi-head latent Mamba architecture, which employs multiple low-dimensional Mamba layers and fully connected layers.
  arXiv  Detail & Related papers  (2025-04-10T02:43:19Z)
• Think Before Recommend: Unleashing the Latent Reasoning Power for Sequential Recommendation [20.965068290049057]
  We propose textbfReaRec, the first inference-time computing framework for recommender systems. ReaRec autoregressively feeds the sequence's last hidden state into the sequential recommender. We introduce two lightweight reasoning-based learning methods, Ensemble Reasoning Learning (ERL) and Progressive Reasoning Learning (PRL)
  arXiv  Detail & Related papers  (2025-03-28T17:59:03Z)
• A Survey of Direct Preference Optimization [103.59317151002693]
  Large Language Models (LLMs) have demonstrated unprecedented generative capabilities. Their alignment with human values remains critical for ensuring helpful and harmless deployments. Direct Preference Optimization (DPO) has recently gained prominence as a streamlined alternative.
  arXiv  Detail & Related papers  (2025-03-12T08:45:15Z)
• Semantic Convergence: Harmonizing Recommender Systems via Two-Stage Alignment and Behavioral Semantic Tokenization [10.47505806629852]
  Large language models (LLMs) are adept at discerning profound user interests from historical behaviors.<n>We propose a novel framework that harmoniously merges traditional recommendation models with the prowess of LLMs.<n>We design a series of specialized supervised learning tasks aimed at aligning collaborative signals with the subtleties of natural language semantics.
  arXiv  Detail & Related papers  (2024-12-18T12:07:58Z)
• Pre-train, Align, and Disentangle: Empowering Sequential Recommendation with Large Language Models [38.15316444108154]
  Sequential recommendation (SR) aims to model the sequential dependencies in users' historical interactions to better capture their evolving interests.<n>Existing SR approaches rely on collaborative data, which leads to limitations such as the cold-start problem and sub-optimal performance.<n>We propose a novel Pre-train, Align, and Disentangle (PAD) paradigm to empower recommendation models with large language models.
  arXiv  Detail & Related papers  (2024-12-05T12:17:56Z)
• Optimizing Sequential Recommendation Models with Scaling Laws and Approximate Entropy [104.48511402784763]
  Performance Law for SR models aims to theoretically investigate and model the relationship between model performance and data quality.<n>We propose Approximate Entropy (ApEn) to assess data quality, presenting a more nuanced approach compared to traditional data quantity metrics.
  arXiv  Detail & Related papers  (2024-11-30T10:56:30Z)
• Hierarchical Reinforcement Learning for Temporal Abstraction of Listwise Recommendation [51.06031200728449]
  We propose a novel framework called mccHRL to provide different levels of temporal abstraction on listwise recommendation. Within the hierarchical framework, the high-level agent studies the evolution of user perception, while the low-level agent produces the item selection policy. Results observe significant performance improvement by our method, compared with several well-known baselines.
  arXiv  Detail & Related papers  (2024-09-11T17:01:06Z)
• Sequential Recommendation via Adaptive Robust Attention with Multi-dimensional Embeddings [7.207685588038045]
  Sequential recommendation models have achieved state-of-the-art performance using self-attention mechanism. Moving beyond only using item ID and positional embeddings leads to a significant accuracy boost when predicting the next item. We introduce a mix-attention mechanism with a layer-wise noise injection (LNI) regularization to improve the model's robustness and generalization.
  arXiv  Detail & Related papers  (2024-09-08T08:27:22Z)
• MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation [61.45986275328629]
  We propose MISSRec, a multi-modal pre-training and transfer learning framework for sequential recommendation. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal user interests. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation.
  arXiv  Detail & Related papers  (2023-08-22T04:06:56Z)
• Contrastive Proposal Extension with LSTM Network for Weakly Supervised Object Detection [52.86681130880647]
  Weakly supervised object detection (WSOD) has attracted more and more attention since it only uses image-level labels and can save huge annotation costs. We propose a new method by comparing the initial proposals and the extension ones to optimize those initial proposals. Experiments on PASCAL VOC 2007, VOC 2012 and MS-COCO datasets show that our method has achieved the state-of-the-art results.
  arXiv  Detail & Related papers  (2021-10-14T16:31:57Z)
• Adversarial and Contrastive Variational Autoencoder for Sequential Recommendation [25.37244686572865]
  We propose a novel method called Adversarial and Contrastive Variational Autoencoder (ACVAE) for sequential recommendation. We first introduce the adversarial training for sequence generation under the Adversarial Variational Bayes framework, which enables our model to generate high-quality latent variables. Besides, when encoding the sequence, we apply a recurrent and convolutional structure to capture global and local relationships in the sequence.
  arXiv  Detail & Related papers  (2021-03-19T09:01:14Z)
• A Generic Network Compression Framework for Sequential Recommender Systems [71.81962915192022]
  Sequential recommender systems (SRS) have become the key technology in capturing user's dynamic interests and generating high-quality recommendations. We propose a compressed sequential recommendation framework, termed as CpRec, where two generic model shrinking techniques are employed. By the extensive ablation studies, we demonstrate that the proposed CpRec can achieve up to 4$sim$8 times compression rates in real-world SRS datasets.
  arXiv  Detail & Related papers  (2020-04-21T08:40:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.