LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems

• URL: http://arxiv.org/abs/2411.01537v1
• Date: Sun, 03 Nov 2024 11:56:00 GMT
• Title: LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems
• Authors: Langming Liu, Xiangyu Zhao, Chi Zhang, Jingtong Gao, Wanyu Wang, Wenqi Fan, Yiqi Wang, Ming He, Zitao Liu, Qing Li,
• Abstract summary: We propose a novel L2-Normalized Linear Attention for the Transformer-based Sequential Recommender Systems (LinRec) We show that LinRec possesses linear complexity while preserving the property of attention mechanisms. Experiments are conducted based on two public benchmark datasets, demonstrating that the combination of LinRec and Transformer models achieves comparable or even superior performance.
• Score: 36.470868461685896
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer models have achieved remarkable success in sequential recommender systems (SRSs). However, computing the attention matrix in traditional dot-product attention mechanisms results in a quadratic complexity with sequence lengths, leading to high computational costs for long-term sequential recommendation. Motivated by the above observation, we propose a novel L2-Normalized Linear Attention for the Transformer-based Sequential Recommender Systems (LinRec), which theoretically improves efficiency while preserving the learning capabilities of the traditional dot-product attention. Specifically, by thoroughly examining the equivalence conditions of efficient attention mechanisms, we show that LinRec possesses linear complexity while preserving the property of attention mechanisms. In addition, we reveal its latent efficiency properties by interpreting the proposed LinRec mechanism through a statistical lens. Extensive experiments are conducted based on two public benchmark datasets, demonstrating that the combination of LinRec and Transformer models achieves comparable or even superior performance than state-of-the-art Transformer-based SRS models while significantly improving time and memory efficiency.

Related papers

• LINA: Linear Autoregressive Image Generative Models with Continuous Tokens [56.80443965097921]
  Autoregressive models with continuous tokens form a promising paradigm for visual generation, especially for text-to-image (T2I) synthesis.<n>We study how to design compute-efficient linear attention within this framework.<n>We present LINA, a simple and compute-efficient T2I model built entirely on linear attention, capable of generating high-fidelity 1024x1024 images from user instructions.
  arXiv  Detail & Related papers  (2026-01-30T06:44:33Z)
• Dissecting Linear Recurrent Models: How Different Gating Strategies Drive Selectivity and Generalization [5.057995083193427]
  Linear recurrent neural networks have emerged as efficient alternatives to the original Transformer's softmax attention mechanism.<n>Existing benchmark tasks are either too simplistic to reveal substantial differences or excessively resource-intensive for experimentation.<n>We introduce SelectivBench, a set of lightweight and customizable synthetic benchmark tasks for systematically evaluating sequence models.
  arXiv  Detail & Related papers  (2026-01-18T21:49:21Z)
• BlossomRec: Block-level Fused Sparse Attention Mechanism for Sequential Recommendations [29.069570226262073]
  Transformer structures have been widely used in sequential recommender systems (SRS)<n>BlossomRec models both long-term and short-term user interests through attention to achieve stable performance across sequences of varying lengths.
  arXiv  Detail & Related papers  (2025-12-15T14:23:57Z)
• DyKAF: Dynamical Kronecker Approximation of the Fisher Information Matrix for Gradient Preconditioning [47.17050585542348]
  We introduce DyKAF (Dynamic Kronalecker Approximation of the Fisher Matrix), which consistently improves the Fisher matrix quality pre-tuning model.<n>Our experiments demonstrate that DyKAF outperforms existing approximations across a range of evaluation metrics.
  arXiv  Detail & Related papers  (2025-11-09T17:48:26Z)
• Why Generate When You Can Transform? Unleashing Generative Attention for Dynamic Recommendation [9.365893765448366]
  Sequential Recommendation (SR) focuses on personalizing user experiences by predicting future preferences based on historical interactions.<n> Transformer models, with their attention mechanisms, have become the dominant architecture in SR tasks.<n>We introduce two generative attention models for SR, each grounded in the principles of Variational Autoencoders (VAE) and Diffusion Models (DMs)
  arXiv  Detail & Related papers  (2025-08-04T04:33:26Z)
• When Relevance Meets Novelty: Dual-Stable Periodic Optimization for Exploratory Recommendation [6.663356205396985]
  Large language models (LLMs) demonstrate potential with their diverse content generation capabilities.<n>Existing LLM-enhanced dual-model frameworks face two major limitations.<n>First, they overlook long-term preferences driven by group identity, leading to biased interest modeling.<n>Second, they suffer from static optimization flaws, as a one-time alignment process fails to leverage incremental user data for closed-loop optimization.
  arXiv  Detail & Related papers  (2025-08-01T09:10:56Z)
• Gated Rotary-Enhanced Linear Attention for Long-term Sequential Recommendation [10.854799326775455]
  We propose a long-term sequential Recommendation model with Gated Rotary Enhanced Linear Attention (RecGRELA)<n>We first propose a Rotary-Enhanced Linear Attention (RELA) module to model long-range dependency within the user's historical information.<n>We then introduce a local short operation to incorporate local preferences and demonstrate the theoretical insight.
  arXiv  Detail & Related papers  (2025-06-16T09:56:10Z)
• LARES: Latent Reasoning for Sequential Recommendation [96.26996622771593]
  We present LARES, a novel and scalable LAtent REasoning framework for Sequential recommendation.<n>Our proposed approach employs a recurrent architecture that allows flexible expansion of reasoning depth without increasing parameter complexity.<n>Our framework exhibits seamless compatibility with existing advanced models, further improving their recommendation performance.
  arXiv  Detail & Related papers  (2025-05-22T16:22:54Z)
• Recurrent Stochastic Configuration Networks with Hybrid Regularization for Nonlinear Dynamics Modelling [3.8719670789415925]
  Recurrent configuration networks (RSCNs) have shown great potential in modelling nonlinear dynamic systems with uncertainties. This paper presents an RSCN with hybrid regularization to enhance both the learning capacity and generalization performance of the network.
  arXiv  Detail & Related papers  (2024-11-26T03:06:39Z)
• Learning Linear Attention in Polynomial Time [115.68795790532289]
  We provide the first results on learnability of single-layer Transformers with linear attention. We show that linear attention may be viewed as a linear predictor in a suitably defined RKHS. We show how to efficiently identify training datasets for which every empirical riskr is equivalent to the linear Transformer.
  arXiv  Detail & Related papers  (2024-10-14T02:41:01Z)
• ELASTIC: Efficient Linear Attention for Sequential Interest Compression [5.689306819772134]
  State-of-the-art sequential recommendation models heavily rely on transformer's attention mechanism. We propose ELASTIC, an Efficient Linear Attention for SequenTial Interest Compression. We conduct extensive experiments on various public datasets and compare it with several strong sequential recommenders.
  arXiv  Detail & Related papers  (2024-08-18T06:41:46Z)
• Behavior-Dependent Linear Recurrent Units for Efficient Sequential Recommendation [18.75561256311228]
  RecBLR is an Efficient Sequential Recommendation Model based on Behavior-Dependent Linear Recurrent Units. Our model significantly enhances user behavior modeling and recommendation performance.
  arXiv  Detail & Related papers  (2024-06-18T13:06:58Z)
• Short-Long Convolutions Help Hardware-Efficient Linear Attention to Focus on Long Sequences [60.489682735061415]
  We propose CHELA, which replaces state space models with short-long convolutions and implements linear attention in a divide-and-conquer manner. Our experiments on the Long Range Arena benchmark and language modeling tasks demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2024-06-12T12:12:38Z)
• EchoMamba4Rec: Harmonizing Bidirectional State Space Models with Spectral Filtering for Advanced Sequential Recommendation [0.0]
  Control theory emphasizes the use of state space models (SSMs) for managing long-range dependencies. We introduce EchoMamba4Rec, a model for predicting user preferences and sequential dependencies based on historical behavior. We show that EchoMamba significantly outperforms existing models, providing more accurate and personalized recommendations.
  arXiv  Detail & Related papers  (2024-06-04T09:07:58Z)
• Fast Dual-Regularized Autoencoder for Sparse Biological Data [65.268245109828]
  We develop a shallow autoencoder for the dual neighborhood-regularized matrix completion problem. We demonstrate the speed and accuracy advantage of our approach over the existing state-of-the-art in predicting drug-target interactions and drug-disease associations.
  arXiv  Detail & Related papers  (2024-01-30T01:28:48Z)
• Associative Transformer [25.1364199879332]
  We propose Associative Transformer (AiT) to enhance the association among sparsely attended input tokens. AiT uses a learnable explicit memory comprising specialized priors that guide bottleneck attentions to facilitate the extraction of diverse localized tokens.
  arXiv  Detail & Related papers  (2023-09-22T13:37:10Z)
• Sequence-to-Sequence Model with Transformer-based Attention Mechanism and Temporal Pooling for Non-Intrusive Load Monitoring [0.0]
  The paper aims to improve the accuracy of Non-Intrusive Load Monitoring (NILM) by using a deep learning-based method. The proposed method uses a Seq2Seq model with a transformer-based attention mechanism to capture the long-term dependencies of NILM data.
  arXiv  Detail & Related papers  (2023-06-08T08:04:56Z)
• Sketching as a Tool for Understanding and Accelerating Self-attention for Long Sequences [52.6022911513076]
  Transformer-based models are not efficient in processing long sequences due to the quadratic space and time complexity of the self-attention modules. We propose Linformer and Informer to reduce the quadratic complexity to linear (modulo logarithmic factors) via low-dimensional projection and row selection. Based on the theoretical analysis, we propose Skeinformer to accelerate self-attention and further improve the accuracy of matrix approximation to self-attention.
  arXiv  Detail & Related papers  (2021-12-10T06:58:05Z)
• Attention that does not Explain Away [54.42960937271612]
  Models based on the Transformer architecture have achieved better accuracy than the ones based on competing architectures for a large set of tasks. A unique feature of the Transformer is its universal application of a self-attention mechanism, which allows for free information flow at arbitrary distances. We propose a doubly-normalized attention scheme that is simple to implement and provides theoretical guarantees for avoiding the "explaining away" effect.
  arXiv  Detail & Related papers  (2020-09-29T21:05:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.