Large Language Models as Conversational Movie Recommenders: A User Study

• URL: http://arxiv.org/abs/2404.19093v1
• Date: Mon, 29 Apr 2024 20:17:06 GMT
• Title: Large Language Models as Conversational Movie Recommenders: A User Study
• Authors: Ruixuan Sun, Xinyi Li, Avinash Akella, Joseph A. Konstan,
• Abstract summary: Large language models (LLMs) offer strong recommendation explainability but lack overall personalization, diversity, and user trust. LLMs show a greater ability to recommend lesser-known or niche movies.
• Score: 3.3636849604467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper explores the effectiveness of using large language models (LLMs) for personalized movie recommendations from users' perspectives in an online field experiment. Our study involves a combination of between-subject prompt and historic consumption assessments, along with within-subject recommendation scenario evaluations. By examining conversation and survey response data from 160 active users, we find that LLMs offer strong recommendation explainability but lack overall personalization, diversity, and user trust. Our results also indicate that different personalized prompting techniques do not significantly affect user-perceived recommendation quality, but the number of movies a user has watched plays a more significant role. Furthermore, LLMs show a greater ability to recommend lesser-known or niche movies. Through qualitative analysis, we identify key conversational patterns linked to positive and negative user interaction experiences and conclude that providing personal context and examples is crucial for obtaining high-quality recommendations from LLMs.

Related papers

• Active Preference-based Learning for Multi-dimensional Personalization [7.349038301460469]
  Large language models (LLMs) have shown remarkable versatility across tasks, but aligning them with individual human preferences remains challenging. We propose an active preference learning framework that uses binary feedback to estimate user preferences across multiple objectives. We validate our approach through theoretical analysis and experiments on language generation tasks, demonstrating its feedback efficiency and effectiveness in personalizing model responses.
  arXiv  Detail & Related papers  (2024-11-01T11:49:33Z)
• Large Language Models as Narrative-Driven Recommenders [0.051205673783866146]
  Large language models (LLMs) have been shown to excel in processing general natural language queries. We compare the performance of 38 open- and closed-source LLMs of various sizes in a movie recommendation setting. Our findings demonstrate the ability of LLMs to generate contextually relevant movie recommendations.
  arXiv  Detail & Related papers  (2024-10-17T14:39:24Z)
• Few-shot Personalization of LLMs with Mis-aligned Responses [40.0349773257245]
  This paper proposes a new approach for a few-shot personalization of large language models (LLMs) Our key idea is to learn a set of personalized prompts for each user by progressively improving the prompts using LLMs. During an iterative process of prompt improvement, we incorporate the contexts of mis-aligned responses by LLMs.
  arXiv  Detail & Related papers  (2024-06-26T18:29:12Z)
• Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs [57.16442740983528]
  In ad-hoc retrieval, evaluation relies heavily on user actions, including implicit feedback. The role of user feedback in annotators' assessment of turns in a conversational perception has been little studied. We focus on how the evaluation of task-oriented dialogue systems ( TDSs) is affected by considering user feedback, explicit or implicit, as provided through the follow-up utterance of a turn being evaluated.
  arXiv  Detail & Related papers  (2024-04-19T16:45:50Z)
• A Large Language Model Enhanced Sequential Recommender for Joint Video and Comment Recommendation [77.42486522565295]
  We propose a novel recommendation approach called LSVCR to jointly conduct personalized video and comment recommendation. Our approach consists of two key components, namely sequential recommendation (SR) model and supplemental large language model (LLM) recommender. In particular, we achieve a significant overall gain of 4.13% in comment watch time.
  arXiv  Detail & Related papers  (2024-03-20T13:14:29Z)
• What Are We Optimizing For? A Human-centric Evaluation of Deep Learning-based Movie Recommenders [12.132920692489911]
  We conduct a human-centric evaluation case study of four leading DL-RecSys models in the movie domain. We test how different DL-RecSys models perform in personalized recommendation generation by conducting survey study with 445 real active users. We find some DL-RecSys models to be superior in recommending novel and unexpected items and weaker in diversity, trustworthiness, transparency, accuracy, and overall user satisfaction.
  arXiv  Detail & Related papers  (2024-01-21T23:56:57Z)
• LLM-Rec: Personalized Recommendation via Prompting Large Language Models [62.481065357472964]
  Large language models (LLMs) have showcased their ability to harness commonsense knowledge and reasoning. Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. This study introduces a novel approach, coined LLM-Rec, which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations.
  arXiv  Detail & Related papers  (2023-07-24T18:47:38Z)
• Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulator to Enhance Dialogue System [65.93577256431125]
  We propose an alternative approach called User-Guided Response Optimization (UGRO) to combine it with a smaller task-oriented dialogue model. This approach uses LLM as annotation-free user simulator to assess dialogue responses, combining them with smaller fine-tuned end-to-end TOD models. Our approach outperforms previous state-of-the-art (SOTA) results.
  arXiv  Detail & Related papers  (2023-06-16T13:04:56Z)
• A Survey on Large Language Models for Recommendation [77.91673633328148]
  Large Language Models (LLMs) have emerged as powerful tools in the field of Natural Language Processing (NLP) This survey presents a taxonomy that categorizes these models into two major paradigms, respectively Discriminative LLM for Recommendation (DLLM4Rec) and Generative LLM for Recommendation (GLLM4Rec)
  arXiv  Detail & Related papers  (2023-05-31T13:51:26Z)
• Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models [115.7508325840751]
  The recent success of large language models (LLMs) has shown great potential to develop more powerful conversational recommender systems (CRSs) In this paper, we embark on an investigation into the utilization of ChatGPT for conversational recommendation, revealing the inadequacy of the existing evaluation protocol. We propose an interactive Evaluation approach based on LLMs named iEvaLM that harnesses LLM-based user simulators.
  arXiv  Detail & Related papers  (2023-05-22T15:12:43Z)
• Recommendation with User Active Disclosing Willingness [20.306413327597603]
  We study a novel recommendation paradigm, where the users are allowed to indicate their "willingness" on disclosing different behaviors. We conduct extensive experiments to demonstrate the effectiveness of our model on balancing the recommendation quality and user disclosing willingness.
  arXiv  Detail & Related papers  (2022-10-25T04:43:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.