Enhancing Item Tokenization for Generative Recommendation through Self-Improvement

• URL: http://arxiv.org/abs/2412.17171v1
• Date: Sun, 22 Dec 2024 21:56:15 GMT
• Title: Enhancing Item Tokenization for Generative Recommendation through Self-Improvement
• Authors: Runjin Chen, Mingxuan Ju, Ngoc Bui, Dimosthenis Antypas, Stanley Cai, Xiaopeng Wu, Leonardo Neves, Zhangyang Wang, Neil Shah, Tong Zhao,
• Abstract summary: Generative recommendation systems are driven by large language models (LLMs) Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens. We propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process.
• Score: 67.94240423434944
• License:
• Abstract: Generative recommendation systems, driven by large language models (LLMs), present an innovative approach to predicting user preferences by modeling items as token sequences and generating recommendations in a generative manner. A critical challenge in this approach is the effective tokenization of items, ensuring that they are represented in a form compatible with LLMs. Current item tokenization methods include using text descriptions, numerical strings, or sequences of discrete tokens. While text-based representations integrate seamlessly with LLM tokenization, they are often too lengthy, leading to inefficiencies and complicating accurate generation. Numerical strings, while concise, lack semantic depth and fail to capture meaningful item relationships. Tokenizing items as sequences of newly defined tokens has gained traction, but it often requires external models or algorithms for token assignment. These external processes may not align with the LLM's internal pretrained tokenization schema, leading to inconsistencies and reduced model performance. To address these limitations, we propose a self-improving item tokenization method that allows the LLM to refine its own item tokenizations during training process. Our approach starts with item tokenizations generated by any external model and periodically adjusts these tokenizations based on the LLM's learned patterns. Such alignment process ensures consistency between the tokenization and the LLM's internal understanding of the items, leading to more accurate recommendations. Furthermore, our method is simple to implement and can be integrated as a plug-and-play enhancement into existing generative recommendation systems. Experimental results on multiple datasets and using various initial tokenization strategies demonstrate the effectiveness of our method, with an average improvement of 8\% in recommendation performance.

Related papers

• Order-agnostic Identifier for Large Language Model-based Generative Recommendation [94.37662915542603]
  Items are assigned identifiers for Large Language Models (LLMs) to encode user history and generate the next item. Existing approaches leverage either token-sequence identifiers, representing items as discrete token sequences, or single-token identifiers, using ID or semantic embeddings. We propose SETRec, which leverages semantic tokenizers to obtain order-agnostic multi-dimensional tokens.
  arXiv  Detail & Related papers  (2025-02-15T15:25:38Z)
• Context-aware Prompt Tuning: Advancing In-Context Learning with Adversarial Methods [69.36397993451742]
  This work introduces Context-aware Prompt Tuning (CPT), a method inspired by ICL, PT, and adversarial attacks. We modify specific context tokens, considering the unique structure of input and output formats. Inspired by adversarial attacks, we adjust the input based on the labels present in the context, focusing on minimizing, rather than maximizing, the loss.
  arXiv  Detail & Related papers  (2024-10-22T17:45:47Z)
• STORE: Streamlining Semantic Tokenization and Generative Recommendation with A Single LLM [59.08493154172207]
  We propose a unified framework to streamline the semantic tokenization and generative recommendation process. We formulate semantic tokenization as a text-to-token task and generative recommendation as a token-to-token task, supplemented by a token-to-text reconstruction task and a text-to-token auxiliary task. All these tasks are framed in a generative manner and trained using a single large language model (LLM) backbone.
  arXiv  Detail & Related papers  (2024-09-11T13:49:48Z)
• Adaptive Draft-Verification for Efficient Large Language Model Decoding [24.347886232342862]
  Large language model (LLM) decoding involves generating a sequence of tokens based on a given context. The typical autoregressive decoding method requires a separate forward pass through the model for each token generated. We introduce ADED, which accelerates LLM decoding without requiring fine-tuning.
  arXiv  Detail & Related papers  (2024-06-27T22:20:39Z)
• TokenRec: Learning to Tokenize ID for LLM-based Generative Recommendation [16.93374578679005]
  TokenRec is a novel framework for tokenizing and retrieving large-scale language models (LLMs) based Recommender Systems (RecSys) Our strategy, Masked Vector-Quantized (MQ) Tokenizer, quantizes the masked user/item representations learned from collaborative filtering into discrete tokens. Our generative retrieval paradigm is designed to efficiently recommend top-$K$ items for users to eliminate the need for auto-regressive decoding and beam search processes.
  arXiv  Detail & Related papers  (2024-06-15T00:07:44Z)
• One Token Can Help! Learning Scalable and Pluggable Virtual Tokens for Retrieval-Augmented Large Language Models [67.49462724595445]
  Retrieval-augmented generation (RAG) is a promising way to improve large language models (LLMs) We propose a novel method that involves learning scalable and pluggable virtual tokens for RAG.
  arXiv  Detail & Related papers  (2024-05-30T03:44:54Z)
• SED: Self-Evaluation Decoding Enhances Large Language Models for Better Generation [35.10931307279044]
  This paper proposes Self-Evaluation Decoding, SED, a decoding method for enhancing model generation. It integrates speculation and evaluation steps into the decoding process, allowing LLMs to make more careful decisions.
  arXiv  Detail & Related papers  (2024-05-26T12:43:18Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.