REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives

• URL: http://arxiv.org/abs/2503.11924v1
• Date: Fri, 14 Mar 2025 23:47:46 GMT
• Title: REGEN: A Dataset and Benchmarks with Natural Language Critiques and Narratives
• Authors: Kun Su, Krishna Sayana, Hubert Pham, James Pine, Yuri Vasilevski, Raghavendra Vasudeva, Marialena Kyriakidi, Liam Hebert, Ambarish Jash, Anushya Subbiah, Sukhdeep Sodhi,
• Abstract summary: We extend the Amazon Product Reviews dataset by inpainting two key natural language features.<n>The narratives include product endorsements, purchase explanations, and summaries of user preferences.
• Score: 4.558818396613368
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper introduces a novel dataset REGEN (Reviews Enhanced with GEnerative Narratives), designed to benchmark the conversational capabilities of recommender Large Language Models (LLMs), addressing the limitations of existing datasets that primarily focus on sequential item prediction. REGEN extends the Amazon Product Reviews dataset by inpainting two key natural language features: (1) user critiques, representing user "steering" queries that lead to the selection of a subsequent item, and (2) narratives, rich textual outputs associated with each recommended item taking into account prior context. The narratives include product endorsements, purchase explanations, and summaries of user preferences. Further, we establish an end-to-end modeling benchmark for the task of conversational recommendation, where models are trained to generate both recommendations and corresponding narratives conditioned on user history (items and critiques). For this joint task, we introduce a modeling framework LUMEN (LLM-based Unified Multi-task Model with Critiques, Recommendations, and Narratives) which uses an LLM as a backbone for critiquing, retrieval and generation. We also evaluate the dataset's quality using standard auto-rating techniques and benchmark it by training both traditional and LLM-based recommender models. Our results demonstrate that incorporating critiques enhances recommendation quality by enabling the recommender to learn language understanding and integrate it with recommendation signals. Furthermore, LLMs trained on our dataset effectively generate both recommendations and contextual narratives, achieving performance comparable to state-of-the-art recommenders and language models.

Related papers

• Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback [50.84142264245052]
  This work introduces the Align-SLM framework to enhance the semantic understanding of textless Spoken Language Models (SLMs) Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO) We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation.
  arXiv  Detail & Related papers  (2024-11-04T06:07:53Z)
• Beyond Retrieval: Generating Narratives in Conversational Recommender Systems [4.73461454584274]
  We introduce a new dataset (REGEN) for natural language generation tasks in conversational recommendations.<n>We establish benchmarks using well-known generative metrics, and perform an automated evaluation of the new dataset using a rater LLM.<n>And to the best of our knowledge, represents the first attempt to analyze the capabilities of LLMs in understanding recommender signals and generating rich narratives.
  arXiv  Detail & Related papers  (2024-10-22T07:53:41Z)
• IDGenRec: LLM-RecSys Alignment with Textual ID Learning [48.018397048791115]
  We propose IDGen, representing each item as a unique, concise, semantically rich, platform-agnostic textual ID. We show that IDGen consistently surpasses existing models in sequential recommendation under standard experimental setting. Results show that the zero-shot performance of the pre-trained foundation model is comparable to or even better than some traditional recommendation models.
  arXiv  Detail & Related papers  (2024-03-27T21:22:37Z)
• Parameter-Efficient Conversational Recommender System as a Language Processing Task [52.47087212618396]
  Conversational recommender systems (CRS) aim to recommend relevant items to users by eliciting user preference through natural language conversation. Prior work often utilizes external knowledge graphs for items' semantic information, a language model for dialogue generation, and a recommendation module for ranking relevant items. In this paper, we represent items in natural language and formulate CRS as a natural language processing task.
  arXiv  Detail & Related papers  (2024-01-25T14:07:34Z)
• LlamaRec: Two-Stage Recommendation using Large Language Models for Ranking [10.671747198171136]
  We propose a two-stage framework using large language models for ranking-based recommendation (LlamaRec) In particular, we use small-scale sequential recommenders to retrieve candidates based on the user interaction history. LlamaRec consistently achieves datasets superior performance in both recommendation performance and efficiency.
  arXiv  Detail & Related papers  (2023-10-25T06:23:48Z)
• ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation [43.270424225285105]
  We focus on adapting and empowering a pure large language model for zero-shot and few-shot recommendation tasks. We propose Retrieval-enhanced Large Language models (ReLLa) for recommendation tasks in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2023-08-22T02:25:04Z)
• Large Language Models as Zero-Shot Conversational Recommenders [52.57230221644014]
  We present empirical studies on conversational recommendation tasks using representative large language models in a zero-shot setting. We construct a new dataset of recommendation-related conversations by scraping a popular discussion website. We observe that even without fine-tuning, large language models can outperform existing fine-tuned conversational recommendation models.
  arXiv  Detail & Related papers  (2023-08-19T15:29:45Z)
• GenRec: Large Language Model for Generative Recommendation [41.22833600362077]
  This paper presents an innovative approach to recommendation systems using large language models (LLMs) based on text data. GenRec uses LLM's understanding ability to interpret context, learn user preferences, and generate relevant recommendation. Our research underscores the potential of LLM-based generative recommendation in revolutionizing the domain of recommendation systems.
  arXiv  Detail & Related papers  (2023-07-02T02:37:07Z)
• Large Language Model Augmented Narrative Driven Recommendations [51.77271767160573]
  Narrative-driven recommendation (NDR) presents an information access problem where users solicit recommendations with verbose descriptions of their preferences and context. NDR lacks abundant training data for models, and current platforms commonly do not support these requests. We use large language models (LLMs) for data augmentation to train NDR models.
  arXiv  Detail & Related papers  (2023-06-04T03:46:45Z)
• Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation [62.019437228000776]
  We present TalkWalk, which generates realistic high-quality conversational data by leveraging encoded expertise in widely available item collections. We generate over one million diverse conversations in a human-collected dataset.
  arXiv  Detail & Related papers  (2023-01-27T01:54:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.