MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation

• URL: http://arxiv.org/abs/2008.08273v2
• Date: Fri, 21 Aug 2020 07:18:14 GMT
• Title: MEANTIME: Mixture of Attention Mechanisms with Multi-temporal Embeddings for Sequential Recommendation
• Authors: Sung Min Cho, Eunhyeok Park, Sungjoo Yoo
• Abstract summary: Self-attention based models have achieved state-of-the-art performance in sequential recommendation task. These models rely on a simple positional embedding to exploit the sequential nature of the user's history. We propose MEANTIME which employs multiple types of temporal embeddings designed to capture various patterns from the user's behavior sequence.
• Score: 12.386304516106854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, self-attention based models have achieved state-of-the-art performance in sequential recommendation task. Following the custom from language processing, most of these models rely on a simple positional embedding to exploit the sequential nature of the user's history. However, there are some limitations regarding the current approaches. First, sequential recommendation is different from language processing in that timestamp information is available. Previous models have not made good use of it to extract additional contextual information. Second, using a simple embedding scheme can lead to information bottleneck since the same embedding has to represent all possible contextual biases. Third, since previous models use the same positional embedding in each attention head, they can wastefully learn overlapping patterns. To address these limitations, we propose MEANTIME (MixturE of AtteNTIon mechanisms with Multi-temporal Embeddings) which employs multiple types of temporal embeddings designed to capture various patterns from the user's behavior sequence, and an attention structure that fully leverages such diversity. Experiments on real-world data show that our proposed method outperforms current state-of-the-art sequential recommendation methods, and we provide an extensive ablation study to analyze how the model gains from the diverse positional information.

Related papers

• On the Regularization of Learnable Embeddings for Time Series Processing [18.069747511100132]
  We investigate methods to regularize the learning of local learnable embeddings for time series processing. We show that methods preventing the co-adaptation of local and global parameters are particularly effective in this context.
  arXiv  Detail & Related papers  (2024-10-18T17:30:20Z)
• Sequential Recommendation on Temporal Proximities with Contrastive Learning and Self-Attention [3.7182810519704095]
  Sequential recommender systems identify user preferences from their past interactions to predict subsequent items optimally. Recent models often neglect similarities in users' actions that occur implicitly among users during analogous timeframes. We propose a sequential recommendation model called TemProxRec, which includes contrastive learning and self-attention methods to consider temporal proximities.
  arXiv  Detail & Related papers  (2024-02-15T08:33:16Z)
• Forecasting large collections of time series: feature-based methods [7.353918137830393]
  When forecasting large collections of time series, two lines of approaches have been developed using time series features. This chapter discusses the state-of-the-art feature-based methods, with reference to open-source software implementations.
  arXiv  Detail & Related papers  (2023-09-25T01:23:02Z)
• Learning Sequence Representations by Non-local Recurrent Neural Memory [61.65105481899744]
  We propose a Non-local Recurrent Neural Memory (NRNM) for supervised sequence representation learning. Our model is able to capture long-range dependencies and latent high-level features can be distilled by our model. Our model compares favorably against other state-of-the-art methods specifically designed for each of these sequence applications.
  arXiv  Detail & Related papers  (2022-07-20T07:26:15Z)
• Multi-scale Attention Flow for Probabilistic Time Series Forecasting [68.20798558048678]
  We propose a novel non-autoregressive deep learning model, called Multi-scale Attention Normalizing Flow(MANF) Our model avoids the influence of cumulative error and does not increase the time complexity. Our model achieves state-of-the-art performance on many popular multivariate datasets.
  arXiv  Detail & Related papers  (2022-05-16T07:53:42Z)
• Self-Attention Neural Bag-of-Features [103.70855797025689]
  We build on the recently introduced 2D-Attention and reformulate the attention learning methodology. We propose a joint feature-temporal attention mechanism that learns a joint 2D attention mask highlighting relevant information.
  arXiv  Detail & Related papers  (2022-01-26T17:54:14Z)
• Learning Dual Dynamic Representations on Time-Sliced User-Item Interaction Graphs for Sequential Recommendation [62.30552176649873]
  We devise a novel Dynamic Representation Learning model for Sequential Recommendation (DRL-SRe) To better model the user-item interactions for characterizing the dynamics from both sides, the proposed model builds a global user-item interaction graph for each time slice. To enable the model to capture fine-grained temporal information, we propose an auxiliary temporal prediction task over consecutive time slices.
  arXiv  Detail & Related papers  (2021-09-24T07:44:27Z)
• From Static to Dynamic Node Embeddings [61.58641072424504]
  We introduce a general framework for leveraging graph stream data for temporal prediction-based applications. Our proposed framework includes novel methods for learning an appropriate graph time-series representation. We find that the top-3 temporal models are always those that leverage the new $epsilon$-graph time-series representation.
  arXiv  Detail & Related papers  (2020-09-21T16:48:29Z)
• Time-based Sequence Model for Personalization and Recommendation Systems [6.484371238475296]
  We develop a novel recommendation model that explicitly incorporates time information. The model relies on an embedding layer and TSL attention-like mechanism with inner products in different vector spaces. We show that it outperforms more complex models with longer sequence length on the Taobao User Behavior dataset.
  arXiv  Detail & Related papers  (2020-08-27T05:46:47Z)
• Predicting Temporal Sets with Deep Neural Networks [50.53727580527024]
  We propose an integrated solution based on the deep neural networks for temporal sets prediction. A unique perspective is to learn element relationship by constructing set-level co-occurrence graph. We design an attention-based module to adaptively learn the temporal dependency of elements and sets.
  arXiv  Detail & Related papers  (2020-06-20T03:29:02Z)
• D\'ej\`a vu: A Contextualized Temporal Attention Mechanism for Sequential Recommendation [34.505472771669744]
  We argue that the influence from the past events on a user's current action should vary over the course of time and under different context. We propose a Contextualized Temporal Attention Mechanism that learns to weigh historical actions' influence on not only what action it is, but also when and how the action took place.
  arXiv  Detail & Related papers  (2020-01-29T20:27:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.