TokenRec: Learning to Tokenize ID for LLM-based Generative Recommendation

• URL: http://arxiv.org/abs/2406.10450v2
• Date: Sun, 18 Aug 2024 07:56:17 GMT
• Title: TokenRec: Learning to Tokenize ID for LLM-based Generative Recommendation
• Authors: Haohao Qu, Wenqi Fan, Zihuai Zhao, Qing Li,
• Abstract summary: TokenRec is a novel framework for tokenizing and retrieving large-scale language models (LLMs) based Recommender Systems (RecSys) Our strategy, Masked Vector-Quantized (MQ) Tokenizer, quantizes the masked user/item representations learned from collaborative filtering into discrete tokens. Our generative retrieval paradigm is designed to efficiently recommend top-$K$ items for users to eliminate the need for auto-regressive decoding and beam search processes.
• Score: 16.93374578679005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There is a growing interest in utilizing large-scale language models (LLMs) to advance next-generation Recommender Systems (RecSys), driven by their outstanding language understanding and in-context learning capabilities. In this scenario, tokenizing (i.e., indexing) users and items becomes essential for ensuring a seamless alignment of LLMs with recommendations. While several studies have made progress in representing users and items through textual contents or latent representations, challenges remain in efficiently capturing high-order collaborative knowledge into discrete tokens that are compatible with LLMs. Additionally, the majority of existing tokenization approaches often face difficulties in generalizing effectively to new/unseen users or items that were not in the training corpus. To address these challenges, we propose a novel framework called TokenRec, which introduces not only an effective ID tokenization strategy but also an efficient retrieval paradigm for LLM-based recommendations. Specifically, our tokenization strategy, Masked Vector-Quantized (MQ) Tokenizer, involves quantizing the masked user/item representations learned from collaborative filtering into discrete tokens, thus achieving a smooth incorporation of high-order collaborative knowledge and a generalizable tokenization of users and items for LLM-based RecSys. Meanwhile, our generative retrieval paradigm is designed to efficiently recommend top-$K$ items for users to eliminate the need for the time-consuming auto-regressive decoding and beam search processes used by LLMs, thus significantly reducing inference time. Comprehensive experiments validate the effectiveness of the proposed methods, demonstrating that TokenRec outperforms competitive benchmarks, including both traditional recommender systems and emerging LLM-based recommender systems.

Related papers

• Self-Calibrated Listwise Reranking with Large Language Models [137.6557607279876]
  Large language models (LLMs) have been employed in reranking tasks through a sequence-to-sequence approach. This reranking paradigm requires a sliding window strategy to iteratively handle larger candidate sets. We propose a novel self-calibrated listwise reranking method, which aims to leverage LLMs to produce global relevance scores for ranking.
  arXiv  Detail & Related papers  (2024-11-07T10:31:31Z)
• Real-Time Personalization for LLM-based Recommendation with Customized In-Context Learning [57.28766250993726]
  This work explores adapting to dynamic user interests without any model updates. Existing Large Language Model (LLM)-based recommenders often lose the in-context learning ability during recommendation tuning. We propose RecICL, which customizes recommendation-specific in-context learning for real-time recommendations.
  arXiv  Detail & Related papers  (2024-10-30T15:48:36Z)
• Beyond Retrieval: Generating Narratives in Conversational Recommender Systems [4.912663905306209]
  We introduce a new dataset (REGEN) for natural language generation tasks in conversational recommendations. We establish benchmarks using well-known generative metrics, and perform an automated evaluation of the new dataset using a rater LLM. And to the best of our knowledge, represents the first attempt to analyze the capabilities of LLMs in understanding recommender signals and generating rich narratives.
  arXiv  Detail & Related papers  (2024-10-22T07:53:41Z)
• STORE: Streamlining Semantic Tokenization and Generative Recommendation with A Single LLM [59.08493154172207]
  We propose a unified framework to streamline the semantic tokenization and generative recommendation process. We formulate semantic tokenization as a text-to-token task and generative recommendation as a token-to-token task, supplemented by a token-to-text reconstruction task and a text-to-token auxiliary task. All these tasks are framed in a generative manner and trained using a single large language model (LLM) backbone.
  arXiv  Detail & Related papers  (2024-09-11T13:49:48Z)
• Improving LLMs for Recommendation with Out-Of-Vocabulary Tokens [51.584024345378005]
  We show how to effectively tokenize users and items in Large Language Models (LLMs)-based recommender systems. We emphasize the role of out-of-vocabulary (OOV) tokens in addition to the in-vocabulary ones. Our proposed framework outperforms existing state-of-the-art methods across various downstream recommendation tasks.
  arXiv  Detail & Related papers  (2024-06-12T17:59:05Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs) We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• Improve Temporal Awareness of LLMs for Sequential Recommendation [61.723928508200196]
  Large language models (LLMs) have demonstrated impressive zero-shot abilities in solving a wide range of general-purpose tasks. LLMs fall short in recognizing and utilizing temporal information, rendering poor performance in tasks that require an understanding of sequential data. We propose three prompting strategies to exploit temporal information within historical interactions for LLM-based sequential recommendation.
  arXiv  Detail & Related papers  (2024-05-05T00:21:26Z)
• RA-Rec: An Efficient ID Representation Alignment Framework for LLM-based Recommendation [9.606111709136675]
  We present RA-Rec, an efficient ID representation framework for LLM-based recommendation. RA-Rec substantially outperforms current state-of-the-art methods, achieving up to 3.0% absolute HitRate@100 improvements.
  arXiv  Detail & Related papers  (2024-02-07T02:14:58Z)
• Representation Learning with Large Language Models for Recommendation [34.46344639742642]
  We propose a model-agnostic framework RLMRec to enhance recommenders with large language models (LLMs)empowered representation learning. RLMRec incorporates auxiliary textual signals, develops a user/item profiling paradigm empowered by LLMs, and aligns the semantic space of LLMs with the representation space of collaborative relational signals.
  arXiv  Detail & Related papers  (2023-10-24T15:51:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.