InteraRec: Screenshot Based Recommendations Using Multimodal Large Language Models

• URL: http://arxiv.org/abs/2403.00822v2
• Date: Sun, 16 Jun 2024 00:40:15 GMT
• Title: InteraRec: Screenshot Based Recommendations Using Multimodal Large Language Models
• Authors: Saketh Reddy Karra, Theja Tulabandhula,
• Abstract summary: We introduce a sophisticated and interactive recommendation framework denoted as InteraRec. InteraRec captures high-frequency screenshots of web pages as users navigate through a website. We demonstrate the effectiveness of InteraRec in providing users with valuable and personalized offerings.
• Score: 0.6926105253992517
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weblogs, comprised of records detailing user activities on any website, offer valuable insights into user preferences, behavior, and interests. Numerous recommendation algorithms, employing strategies such as collaborative filtering, content-based filtering, and hybrid methods, leverage the data mined through these weblogs to provide personalized recommendations to users. Despite the abundance of information available in these weblogs, identifying and extracting pertinent information and key features from them necessitate extensive engineering endeavors. The intricate nature of the data also poses a challenge for interpretation, especially for non-experts. In this study, we introduce a sophisticated and interactive recommendation framework denoted as InteraRec, which diverges from conventional approaches that exclusively depend on weblogs for recommendation generation. InteraRec framework captures high-frequency screenshots of web pages as users navigate through a website. Leveraging state-of-the-art multimodal large language models (MLLMs), it extracts valuable insights into user preferences from these screenshots by generating a textual summary based on predefined keywords. Subsequently, an LLM-integrated optimization setup utilizes this summary to generate tailored recommendations. Through our experiments, we demonstrate the effectiveness of InteraRec in providing users with valuable and personalized offerings. Furthermore, we explore the integration of session-based recommendation systems into the InteraRec framework, aiming to enhance its overall performance. Finally, we curate a new dataset comprising of screenshots from product web pages on the Amazon website for the validation of the InteraRec framework. Detailed experiments demonstrate the efficacy of the InteraRec framework in delivering valuable and personalized recommendations tailored to individual user preferences.

Related papers

• UserSumBench: A Benchmark Framework for Evaluating User Summarization Approaches [25.133460380551327]
  Large language models (LLMs) have shown remarkable capabilities in generating user summaries from a long list of raw user activity data. These summaries capture essential user information such as preferences and interests, and are invaluable for personalization applications. However, the development of new summarization techniques is hindered by the lack of ground-truth labels, the inherent subjectivity of user summaries, and human evaluation.
  arXiv  Detail & Related papers  (2024-08-30T01:56:57Z)
• Personalized Federated Collaborative Filtering: A Variational AutoEncoder Approach [49.63614966954833]
  Federated Collaborative Filtering (FedCF) is an emerging field focused on developing a new recommendation framework with preserving privacy. This paper proposes a novel personalized FedCF method by preserving users' personalized information into a latent variable and a neural model simultaneously. To effectively train the proposed framework, we model the problem as a specialized Variational AutoEncoder (VAE) task by integrating user interaction vector reconstruction with missing value prediction.
  arXiv  Detail & Related papers  (2024-08-16T05:49:14Z)
• EmbSum: Leveraging the Summarization Capabilities of Large Language Models for Content-Based Recommendations [38.44534579040017]
  We introduce EmbSum, a framework that enables offline pre-computations of users and candidate items. The model's ability to generate summaries of user interests serves as a valuable by-product, enhancing its usefulness for personalized content recommendations.
  arXiv  Detail & Related papers  (2024-05-19T04:31:54Z)
• Knowledge Adaptation from Large Language Model to Recommendation for Practical Industrial Application [54.984348122105516]
  Large Language Models (LLMs) pretrained on massive text corpus presents a promising avenue for enhancing recommender systems. We propose an Llm-driven knowlEdge Adaptive RecommeNdation (LEARN) framework that synergizes open-world knowledge with collaborative knowledge.
  arXiv  Detail & Related papers  (2024-05-07T04:00:30Z)
• A Large Language Model Enhanced Sequential Recommender for Joint Video and Comment Recommendation [77.42486522565295]
  We propose a novel recommendation approach called LSVCR to jointly conduct personalized video and comment recommendation. Our approach consists of two key components, namely sequential recommendation (SR) model and supplemental large language model (LLM) recommender. In particular, we achieve a significant overall gain of 4.13% in comment watch time.
  arXiv  Detail & Related papers  (2024-03-20T13:14:29Z)
• SPAR: Personalized Content-Based Recommendation via Long Engagement Attention [43.04717491985609]
  Leveraging users' long engagement histories is essential for personalized content recommendations. We introduce a content-based recommendation framework, SPAR, which effectively tackles the challenges of holistic user interest extraction. Our framework outperforms existing state-of-the-art (SoTA) methods.
  arXiv  Detail & Related papers  (2024-02-16T10:36:38Z)
• Reformulating Sequential Recommendation: Learning Dynamic User Interest with Content-enriched Language Modeling [18.297332953450514]
  We propose LANCER, which leverages the semantic understanding capabilities of pre-trained language models to generate personalized recommendations. Our approach bridges the gap between language models and recommender systems, resulting in more human-like recommendations.
  arXiv  Detail & Related papers  (2023-09-19T08:54:47Z)
• MISSRec: Pre-training and Transferring Multi-modal Interest-aware Sequence Representation for Recommendation [61.45986275328629]
  We propose MISSRec, a multi-modal pre-training and transfer learning framework for sequential recommendation. On the user side, we design a Transformer-based encoder-decoder model, where the contextual encoder learns to capture the sequence-level multi-modal user interests. On the candidate item side, we adopt a dynamic fusion module to produce user-adaptive item representation.
  arXiv  Detail & Related papers  (2023-08-22T04:06:56Z)
• iFacetSum: Coreference-based Interactive Faceted Summarization for Multi-Document Exploration [63.272359227081836]
  iFacetSum integrates interactive summarization together with faceted search. Fine-grained facets are automatically produced based on cross-document coreference pipelines.
  arXiv  Detail & Related papers  (2021-09-23T20:01:11Z)
• Reward Constrained Interactive Recommendation with Natural Language Feedback [158.8095688415973]
  We propose a novel constraint-augmented reinforcement learning (RL) framework to efficiently incorporate user preferences over time. Specifically, we leverage a discriminator to detect recommendations violating user historical preference. Our proposed framework is general and is further extended to the task of constrained text generation.
  arXiv  Detail & Related papers  (2020-05-04T16:23:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.