Real-Time Personalization with Simple Transformers

• URL: http://arxiv.org/abs/2503.00608v1
• Date: Sat, 01 Mar 2025 20:29:33 GMT
• Title: Real-Time Personalization with Simple Transformers
• Authors: Lin An, Andrew A. Li, Vaisnavi Nemala, Gabriel Visotsky,
• Abstract summary: We show that simple transformers are capable of capturing complex user preferences.<n>We then develop an algorithm that enables fast optimization of recommendation tasks based on simple transformers.<n>Our algorithm achieves near-optimal performance in sub-linear time.
• Score: 5.974778743092437
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Real-time personalization has advanced significantly in recent years, with platforms utilizing machine learning models to predict user preferences based on rich behavioral data on each individual user. Traditional approaches usually rely on embedding-based machine learning models to capture user preferences, and then reduce the final optimization task to nearest-neighbors, which can be performed extremely fast. However, these models struggle to capture complex user behaviors, which are essential for making accurate recommendations. Transformer-based models, on the other hand, are known for their practical ability to model sequential behaviors, and hence have been intensively used in personalization recently to overcome these limitations. However, optimizing recommendations under transformer-based models is challenging due to their complicated architectures. In this paper, we address this challenge by considering a specific class of transformers, showing its ability to represent complex user preferences, and developing efficient algorithms for real-time personalization. We focus on a particular set of transformers, called simple transformers, which contain a single self-attention layer. We show that simple transformers are capable of capturing complex user preferences. We then develop an algorithm that enables fast optimization of recommendation tasks based on simple transformers. Our algorithm achieves near-optimal performance in sub-linear time. Finally, we demonstrate the effectiveness of our approach through an empirical study on datasets from Spotify and Trivago. Our experiment results show that (1) simple transformers can model/predict user preferences substantially more accurately than non-transformer models and nearly as accurately as more complex transformers, and (2) our algorithm completes simple-transformer-based recommendation tasks quickly and effectively.

Related papers

• Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysis [63.66763657191476]
  We show that efficient numerical training and inference algorithms as low-rank computation have impressive performance for learning Transformer-based adaption. We analyze how magnitude-based models affect generalization while improving adaption. We conclude that proper magnitude-based has a slight on the testing performance.
  arXiv  Detail & Related papers  (2024-06-24T23:00:58Z)
• Linear attention is (maybe) all you need (to understand transformer optimization) [55.81555204646486]
  We make progress towards understanding the subtleties of training Transformers by studying a simple yet canonicalized shallow Transformer model. Most importantly, we observe that our proposed linearized models can reproduce several prominent aspects of Transformer training dynamics.
  arXiv  Detail & Related papers  (2023-10-02T10:48:42Z)
• ConvFormer: Revisiting Transformer for Sequential User Modeling [23.650635274170828]
  We re-examine Transformer-like architectures aiming to advance state-of-the-art performance. We identify three essential criteria for devising efficient sequential user models. We introduce ConvFormer, a simple but powerful modification to the Transformer architecture that meets these criteria.
  arXiv  Detail & Related papers  (2023-08-05T17:33:17Z)
• Emergent Agentic Transformer from Chain of Hindsight Experience [96.56164427726203]
  We show that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches. This is the first time that a simple transformer-based model performs competitively with both temporal-difference and imitation-learning-based approaches.
  arXiv  Detail & Related papers  (2023-05-26T00:43:02Z)
• Preference Transformer: Modeling Human Preferences using Transformers for RL [165.33887165572128]
  Preference Transformer is a neural architecture that models human preferences using transformers. We show that Preference Transformer can solve a variety of control tasks using real human preferences, while prior approaches fail to work.
  arXiv  Detail & Related papers  (2023-03-02T04:24:29Z)
• Mnemosyne: Learning to Train Transformers with Transformers [18.36543176998175]
  We show that Mnemosyne can successfully train Transformers while using simple meta-training strategies that require minimal computational resources. Mnemosyne provides space comparable complexity to that its hand-designed first-order counterparts, which allows it to scale to training larger sets of parameters.
  arXiv  Detail & Related papers  (2023-02-02T14:40:28Z)
• Finetuning Pretrained Transformers into RNNs [81.72974646901136]
  Transformers have outperformed recurrent neural networks (RNNs) in natural language generation. A linear-complexity recurrent variant has proven well suited for autoregressive generation. This work aims to convert a pretrained transformer into its efficient recurrent counterpart.
  arXiv  Detail & Related papers  (2021-03-24T10:50:43Z)
• The Cascade Transformer: an Application for Efficient Answer Sentence Selection [116.09532365093659]
  We introduce the Cascade Transformer, a technique to adapt transformer-based models into a cascade of rankers. When compared to a state-of-the-art transformer model, our approach reduces computation by 37% with almost no impact on accuracy.
  arXiv  Detail & Related papers  (2020-05-05T23:32:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.