PMG : Personalized Multimodal Generation with Large Language Models

• URL: http://arxiv.org/abs/2404.08677v1
• Date: Sun, 7 Apr 2024 03:05:57 GMT
• Title: PMG : Personalized Multimodal Generation with Large Language Models
• Authors: Xiaoteng Shen, Rui Zhang, Xiaoyan Zhao, Jieming Zhu, Xi Xiao,
• Abstract summary: This paper proposes the first method for personalized multimodal generation using large language models (LLMs) It showcases its applications and validates its performance via an extensive experimental study on two datasets. PMG has a significant improvement on personalization for up to 8% in terms of LPIPS while retaining the accuracy of generation.
• Score: 20.778869086174137
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The emergence of large language models (LLMs) has revolutionized the capabilities of text comprehension and generation. Multi-modal generation attracts great attention from both the industry and academia, but there is little work on personalized generation, which has important applications such as recommender systems. This paper proposes the first method for personalized multimodal generation using LLMs, showcases its applications and validates its performance via an extensive experimental study on two datasets. The proposed method, Personalized Multimodal Generation (PMG for short) first converts user behaviors (e.g., clicks in recommender systems or conversations with a virtual assistant) into natural language to facilitate LLM understanding and extract user preference descriptions. Such user preferences are then fed into a generator, such as a multimodal LLM or diffusion model, to produce personalized content. To capture user preferences comprehensively and accurately, we propose to let the LLM output a combination of explicit keywords and implicit embeddings to represent user preferences. Then the combination of keywords and embeddings are used as prompts to condition the generator. We optimize a weighted sum of the accuracy and preference scores so that the generated content has a good balance between them. Compared to a baseline method without personalization, PMG has a significant improvement on personalization for up to 8% in terms of LPIPS while retaining the accuracy of generation.

Related papers

• MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time [50.41806216615488]
  Large Language Models (LLMs) acquire extensive knowledge and remarkable abilities from extensive text corpora. To make LLMs more usable, aligning them with human preferences is essential. We propose an effective method, textbf MetaAlign, which aims to help LLMs dynamically align with various explicit or implicit preferences specified at inference time.
  arXiv  Detail & Related papers  (2024-10-18T05:31:13Z)
• Retrieval-Augmented Personalization for Multimodal Large Language Models [53.304699445700926]
  We introduce the Retrieval Augmented Personalization (RAP) framework for MLLMs' personalization. RAP allows real-time concept editing via updating the external database. RAP-MLLMs can generalize to infinite visual concepts without additional finetuning.
  arXiv  Detail & Related papers  (2024-10-17T09:10:26Z)
• LLMs + Persona-Plug = Personalized LLMs [41.60364110693824]
  Personalization plays a critical role in numerous language tasks and applications, since users with the same requirements may prefer diverse outputs based on their individual interests. This has led to the development of various personalized approaches aimed at adapting large language models (LLMs) to generate customized outputs aligned with user preferences. We propose a novel personalized LLM model, ours. It constructs a user-specific embedding for each individual by modeling all her historical contexts through a lightweight plug-in user embedder module.
  arXiv  Detail & Related papers  (2024-09-18T11:54:45Z)
• Laser: Parameter-Efficient LLM Bi-Tuning for Sequential Recommendation with Collaborative Information [76.62949982303532]
  We propose a parameter-efficient Large Language Model Bi-Tuning framework for sequential recommendation with collaborative information (Laser) In our Laser, the prefix is utilized to incorporate user-item collaborative information and adapt the LLM to the recommendation task, while the suffix converts the output embeddings of the LLM from the language space to the recommendation space for the follow-up item recommendation. M-Former is a lightweight MoE-based querying transformer that uses a set of query experts to integrate diverse user-specific collaborative information encoded by frozen ID-based sequential recommender systems.
  arXiv  Detail & Related papers  (2024-09-03T04:55:03Z)
• Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation [21.281471662696372]
  We propose the Multimodal Large Language Model-enhanced Multimodaln Sequential Recommendation (MLLM-MSR) model. To capture the dynamic user preference, we design a two-stage user preference summarization method. We then employ a recurrent user preference summarization generation paradigm to capture the dynamic changes in user preferences.
  arXiv  Detail & Related papers  (2024-08-19T04:44:32Z)
• Orchestrating LLMs with Different Personalizations [28.344891363780576]
  This paper presents a novel approach to aligning large language models (LLMs) with individual human preferences. Given stated preferences along multiple dimensions, such as helpfulness, conciseness, or humor, the goal is to create an LLM without re-training that best adheres to this specification. Starting from specialized expert LLMs, each trained for one particular preference dimension, we propose a black-box method that merges their outputs on a per-token level.
  arXiv  Detail & Related papers  (2024-07-04T22:55:02Z)
• Multi-Reference Preference Optimization for Large Language Models [56.84730239046117]
  We introduce a novel closed-form formulation for direct preference optimization using multiple reference models. The resulting algorithm, Multi-Reference Preference Optimization (MRPO), leverages broader prior knowledge from diverse reference models. Our experiments demonstrate that LLMs finetuned with MRPO generalize better in various preference data, regardless of data scarcity or abundance.
  arXiv  Detail & Related papers  (2024-05-26T00:29:04Z)
• Doing Personal LAPS: LLM-Augmented Dialogue Construction for Personalized Multi-Session Conversational Search [9.243535345193711]
  Our method uses large language models to guide a single human worker in generating personalized dialogues. LAPS can collect large-scale, human-written, multi-session, and multi-domain conversations. Our results show that responses generated explicitly using extracted preferences better match user's actual preferences.
  arXiv  Detail & Related papers  (2024-05-06T13:53:03Z)
• Optimization Methods for Personalizing Large Language Models through Retrieval Augmentation [23.174810143027234]
  This paper studies retrieval-augmented approaches for personalizing large language models (LLMs) We propose the first attempt to optimize the retrieval models that deliver a limited number of personal documents to large language models for the purpose of personalized generation.
  arXiv  Detail & Related papers  (2024-04-09T02:58:05Z)
• RecExplainer: Aligning Large Language Models for Explaining Recommendation Models [50.74181089742969]
  Large language models (LLMs) have demonstrated remarkable intelligence in understanding, reasoning, and instruction following. This paper presents the initial exploration of using LLMs as surrogate models to explain black-box recommender models. To facilitate an effective alignment, we introduce three methods: behavior alignment, intention alignment, and hybrid alignment.
  arXiv  Detail & Related papers  (2023-11-18T03:05:43Z)
• LLM-Rec: Personalized Recommendation via Prompting Large Language Models [62.481065357472964]
  Large language models (LLMs) have showcased their ability to harness commonsense knowledge and reasoning. Recent advances in large language models (LLMs) have showcased their remarkable ability to harness commonsense knowledge and reasoning. This study introduces a novel approach, coined LLM-Rec, which incorporates four distinct prompting strategies of text enrichment for improving personalized text-based recommendations.
  arXiv  Detail & Related papers  (2023-07-24T18:47:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.