LightLM: A Lightweight Deep and Narrow Language Model for Generative Recommendation

• URL: http://arxiv.org/abs/2310.17488v2
• Date: Mon, 30 Oct 2023 02:50:17 GMT
• Title: LightLM: A Lightweight Deep and Narrow Language Model for Generative Recommendation
• Authors: Kai Mei, Yongfeng Zhang
• Abstract summary: LightLM is a lightweight Transformer-based language model for generative recommendation. LightLM tackles the issue by introducing a light-weight deep and narrow Transformer architecture. We also show that our devised user and item ID indexing methods, i.e., Spectral Collaborative Indexing (SCI) and Graph Collaborative Indexing (GCI), enables the deep and narrow Transformer architecture to outperform large-scale language models for recommendation.
• Score: 45.00339682494516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper presents LightLM, a lightweight Transformer-based language model for generative recommendation. While Transformer-based generative modeling has gained importance in various AI sub-fields such as NLP and vision, generative recommendation is still in its infancy due to its unique demand on personalized generative modeling. Existing works on generative recommendation often use NLP-oriented Transformer architectures such as T5, GPT, LLaMA and M6, which are heavy-weight and are not specifically designed for recommendation tasks. LightLM tackles the issue by introducing a light-weight deep and narrow Transformer architecture, which is specifically tailored for direct generation of recommendation items. This structure is especially apt for straightforward generative recommendation and stems from the observation that language model does not have to be too wide for this task, as the input predominantly consists of short tokens that are well-suited for the model's capacity. We also show that our devised user and item ID indexing methods, i.e., Spectral Collaborative Indexing (SCI) and Graph Collaborative Indexing (GCI), enables the deep and narrow Transformer architecture to outperform large-scale language models for recommendation. Besides, to address the hallucination problem of generating items as output, we propose the constrained generation process for generative recommenders. Experiments on real-world datasets show that LightLM outperforms various competitive baselines in terms of both recommendation accuracy and efficiency. The code can be found at https://github.com/dongyuanjushi/LightLM.

Related papers

• Inductive Generative Recommendation via Retrieval-based Speculation [26.70518822003545]
  Generative recommendation (GR) is an emerging paradigm that tokenizes items into discrete tokens and learns to autoregressively generate the next tokens as predictions. In this paper, we propose SpecGR, a plug-and-play framework that enables GR models to recommend new items in an inductive setting.
  arXiv  Detail & Related papers  (2024-10-03T19:32:32Z)
• Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models [79.41139393080736]
  Large language models (LLMs) have rapidly advanced and demonstrated impressive capabilities. In-Context Learning (ICL) and. Efficient Fine-Tuning (PEFT) are currently two mainstream methods for augmenting. LLMs to downstream tasks. We propose Reference Trustable Decoding (RTD), a paradigm that allows models to quickly adapt to new tasks without fine-tuning.
  arXiv  Detail & Related papers  (2024-09-30T10:48:20Z)
• Laser: Parameter-Efficient LLM Bi-Tuning for Sequential Recommendation with Collaborative Information [76.62949982303532]
  We propose a parameter-efficient Large Language Model Bi-Tuning framework for sequential recommendation with collaborative information (Laser) In our Laser, the prefix is utilized to incorporate user-item collaborative information and adapt the LLM to the recommendation task, while the suffix converts the output embeddings of the LLM from the language space to the recommendation space for the follow-up item recommendation. M-Former is a lightweight MoE-based querying transformer that uses a set of query experts to integrate diverse user-specific collaborative information encoded by frozen ID-based sequential recommender systems.
  arXiv  Detail & Related papers  (2024-09-03T04:55:03Z)
• PMG : Personalized Multimodal Generation with Large Language Models [20.778869086174137]
  This paper proposes the first method for personalized multimodal generation using large language models (LLMs) It showcases its applications and validates its performance via an extensive experimental study on two datasets. PMG has a significant improvement on personalization for up to 8% in terms of LPIPS while retaining the accuracy of generation.
  arXiv  Detail & Related papers  (2024-04-07T03:05:57Z)
• IDGenRec: LLM-RecSys Alignment with Textual ID Learning [48.018397048791115]
  We propose IDGen, representing each item as a unique, concise, semantically rich, platform-agnostic textual ID. We show that IDGen consistently surpasses existing models in sequential recommendation under standard experimental setting. Results show that the zero-shot performance of the pre-trained foundation model is comparable to or even better than some traditional recommendation models.
  arXiv  Detail & Related papers  (2024-03-27T21:22:37Z)
• Recommender AI Agent: Integrating Large Language Models for Interactive Recommendations [53.76682562935373]
  We introduce an efficient framework called textbfInteRecAgent, which employs LLMs as the brain and recommender models as tools. InteRecAgent achieves satisfying performance as a conversational recommender system, outperforming general-purpose LLMs.
  arXiv  Detail & Related papers  (2023-08-31T07:36:44Z)
• GenRec: Large Language Model for Generative Recommendation [41.22833600362077]
  This paper presents an innovative approach to recommendation systems using large language models (LLMs) based on text data. GenRec uses LLM's understanding ability to interpret context, learn user preferences, and generate relevant recommendation. Our research underscores the potential of LLM-based generative recommendation in revolutionizing the domain of recommendation systems.
  arXiv  Detail & Related papers  (2023-07-02T02:37:07Z)
• Extrapolating Multilingual Understanding Models as Multilingual Generators [82.1355802012414]
  This paper explores methods to empower multilingual understanding models the generation abilities to get a unified model. We propose a textbfSemantic-textbfGuided textbfAlignment-then-Denoising (SGA) approach to adapt an encoder to a multilingual generator with a small number of new parameters.
  arXiv  Detail & Related papers  (2023-05-22T15:33:21Z)
• Guiding Large Language Models via Directional Stimulus Prompting [114.84930073977672]
  We introduce Directional Stimulus Prompting, a novel framework for guiding black-box large language models (LLMs) toward specific desired outputs. Instead of directly adjusting LLMs, our method employs a small tunable policy model to generate an auxiliary directional stimulus prompt for each input instance.
  arXiv  Detail & Related papers  (2023-02-22T17:44:15Z)
• N-Grammer: Augmenting Transformers with latent n-grams [35.39961549040385]
  We propose a simple yet effective modification to the Transformer architecture inspired by the literature in statistical language modeling, by augmenting the model with n-grams that are constructed from a discrete latent representation of the text sequence. We evaluate our model, the N-Grammer on language modeling on the C4 data-set as well as text classification on the SuperGLUE data-set, and find that it outperforms several strong baselines such as the Transformer and the Primer.
  arXiv  Detail & Related papers  (2022-07-13T17:18:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.