Learning Positional Attention for Sequential Recommendation

• URL: http://arxiv.org/abs/2407.02793v3
• Date: Tue, 26 Nov 2024 06:34:55 GMT
• Title: Learning Positional Attention for Sequential Recommendation
• Abstract summary: We introduce novel attention models that directly learn positional relations. Our proposed models, textbfPARec and textbfFPARec, outperform previous self-attention-based approaches.
• Score: 21.193999811562414
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-attention-based networks have achieved remarkable performance in sequential recommendation tasks. A crucial component of these models is positional encoding. In this study, we delve into the learned positional embedding, demonstrating that it often captures the distance between tokens. Building on this insight, we introduce novel attention models that directly learn positional relations. Extensive experiments reveal that our proposed models, \textbf{PARec} and \textbf{FPARec} outperform previous self-attention-based approaches. The code can be found here: https://github.com/NetEase-Media/FPARec.

Related papers

• Attention, Anomalies! Handling Attention Layers in Unsupervised Federated Outlier Detection [5.092008988231824]
  We analyze the intricacies of the architecture behind Memory Augmented Autoencoders (MemAE)<n>We propose novel, guided approaches to effectively aggregate these models in federated scenarios.<n>This approach improves the performance of even very shallow autoencoders, allowing them to be used in resource constrained environments.
  arXiv  Detail & Related papers  (2026-08-05T12:23:13Z)
• CAPE: A CLIP-Aware Pointing Ensemble of Complementary Heatmap Cues for Embodied Reference Understanding [56.30142869506262]
  Embodied Reference Understanding involves predicting the object that a person in the scene is referring to through both pointing gesture and language.<n>We propose a dual-model framework, where one model learns from the head-to-fingertip direction and the other from the wrist-to-fingertip direction.<n>We validate our approach through extensive experiments and analysis on the benchmark YouRefIt dataset, achieving an improvement of approximately 4 mAP at the 0.25 IoU threshold.
  arXiv  Detail & Related papers  (2025-07-29T15:00:21Z)
• Long-Sequence Recommendation Models Need Decoupled Embeddings [49.410906935283585]
  We identify and characterize a neglected deficiency in existing long-sequence recommendation models. A single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. We propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are learned separately to fully decouple attention and representation.
  arXiv  Detail & Related papers  (2024-10-03T15:45:15Z)
• Sequential Recommendation via Adaptive Robust Attention with Multi-dimensional Embeddings [7.207685588038045]
  Sequential recommendation models have achieved state-of-the-art performance using self-attention mechanism. Moving beyond only using item ID and positional embeddings leads to a significant accuracy boost when predicting the next item. We introduce a mix-attention mechanism with a layer-wise noise injection (LNI) regularization to improve the model's robustness and generalization.
  arXiv  Detail & Related papers  (2024-09-08T08:27:22Z)
• Unsupervised Keypoints from Pretrained Diffusion Models [31.147785019795347]
  We leverage the emergent knowledge within text-to-image diffusion models, towards more robust unsupervised keypoints. Our core idea is to find text embeddings that would cause the generative model to consistently attend to compact regions in images. We validate our performance on multiple datasets: the CelebA, CUB-200-2011, Tai-Chi-HD, DeepFashion, and Human3.6m datasets.
  arXiv  Detail & Related papers  (2023-11-29T19:43:38Z)
• ResMatch: Residual Attention Learning for Local Feature Matching [51.07496081296863]
  We rethink cross- and self-attention from the viewpoint of traditional feature matching and filtering. We inject the similarity of descriptors and relative positions into cross- and self-attention score. We mine intra- and inter-neighbors according to the similarity of descriptors and relative positions.
  arXiv  Detail & Related papers  (2023-07-11T11:32:12Z)
• DenseDINO: Boosting Dense Self-Supervised Learning with Token-Based Point-Level Consistency [12.881617910150688]
  We propose a transformer framework for self-supervised learning called DenseDINO to learn dense visual representations. Specifically, DenseDINO introduces some extra input tokens called reference tokens to match the point-level features with the position prior. Compared with the vanilla DINO, our approach obtains competitive performance when evaluated on classification in ImageNet.
  arXiv  Detail & Related papers  (2023-06-06T15:04:45Z)
• Bootstrap Your Own Correspondences [15.715143016999695]
  BYOC is a self-supervised approach that learns visual and geometric features from RGB-D video without relying on ground-truth pose or correspondence. We evaluate our approach on indoor scene datasets and find that our method outperforms traditional and learned descriptors.
  arXiv  Detail & Related papers  (2021-06-01T17:59:08Z)
• Online Multiple Object Tracking with Cross-Task Synergy [120.70085565030628]
  We propose a novel unified model with synergy between position prediction and embedding association. The two tasks are linked by temporal-aware target attention and distractor attention, as well as identity-aware memory aggregation model.
  arXiv  Detail & Related papers  (2021-04-01T10:19:40Z)
• Revisiting Deep Local Descriptor for Improved Few-Shot Classification [56.74552164206737]
  We show how one can improve the quality of embeddings by leveraging textbfDense textbfClassification and textbfAttentive textbfPooling. We suggest to pool feature maps by applying attentive pooling instead of the widely used global average pooling (GAP) to prepare embeddings for few-shot classification.
  arXiv  Detail & Related papers  (2021-03-30T00:48:28Z)
• Universal Representation Learning from Multiple Domains for Few-shot Classification [41.821234589075445]
  We propose to learn a single set of universal deep representations by distilling knowledge of multiple separately trained networks. We show that the universal representations can be further refined for previously unseen domains by an efficient adaptation step.
  arXiv  Detail & Related papers  (2021-03-25T13:49:12Z)
• S^3-Rec: Self-Supervised Learning for Sequential Recommendation with Mutual Information Maximization [104.87483578308526]
  We propose the model S3-Rec, which stands for Self-Supervised learning for Sequential Recommendation. For our task, we devise four auxiliary self-supervised objectives to learn the correlations among attribute, item, subsequence, and sequence. Extensive experiments conducted on six real-world datasets demonstrate the superiority of our proposed method over existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-08-18T11:44:10Z)
• Relation-Guided Representation Learning [53.60351496449232]
  We propose a new representation learning method that explicitly models and leverages sample relations. Our framework well preserves the relations between samples. By seeking to embed samples into subspace, we show that our method can address the large-scale and out-of-sample problem.
  arXiv  Detail & Related papers  (2020-07-11T10:57:45Z)
• Unsupervised Learning of Video Representations via Dense Trajectory Clustering [86.45054867170795]
  This paper addresses the task of unsupervised learning of representations for action recognition in videos. We first propose to adapt two top performing objectives in this class - instance recognition and local aggregation. We observe promising performance, but qualitative analysis shows that the learned representations fail to capture motion patterns.
  arXiv  Detail & Related papers  (2020-06-28T22:23:03Z)
• Attention improves concentration when learning node embeddings [1.2233362977312945]
  Given nodes labelled with search query text, we want to predict links to related queries that share products. Experiments with a range of deep neural architectures show that simple feedforward networks with an attention mechanism perform best for learning embeddings. We propose an analytically tractable model of query generation, AttEST, that views both products and the query text as vectors embedded in a latent space.
  arXiv  Detail & Related papers  (2020-06-11T21:21:12Z)
• Active Learning for Coreference Resolution using Discrete Annotation [76.36423696634584]
  We improve upon pairwise annotation for active learning in coreference resolution. We ask annotators to identify mention antecedents if a presented mention pair is deemed not coreferent. In experiments with existing benchmark coreference datasets, we show that the signal from this additional question leads to significant performance gains per human-annotation hour.
  arXiv  Detail & Related papers  (2020-04-28T17:17:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.